Repository navigation
feat(storage): reduce persistent disk usage with lossless cold-session compression #6109
Description
Problem
Long-running Maka sessions retain large amounts of persisted conversation and tool-result JSON. Much of this history is no longer being written, but it continues to occupy the same disk space as active sessions. Deleting history is an undesirable workaround when users still need complete transcripts, replay, and tool evidence.
We would like to reduce persistent storage usage while preserving the complete stored data and existing read behavior.
Desired outcome
Introduce hot/cold storage for persisted session content, with lossless compression for cold data:
- Treat a Session as cold after 7 days without an actual local content write. Reading history does not reset this clock; imported/restored content starts a new local cooling period.
- Initially cover completed text, thinking, tool-call arguments, and complete persisted tool responses, including signatures and provider metadata. Keep streaming partials, indexes, and control state directly accessible.
- Compress in bounded, low-priority background work after Host readiness, yielding to execution and frontend reads. Keep original data when compression has insufficient net benefit or resources are constrained.
- Read compressed and uncompressed records transparently. Continuing a conversation appends ordinary records while existing compressed history stays compressed.
- Use a versioned representation with corruption checks, safe publication, and bounded restoration to the original bytes. Future codec/policy changes must remain compatible with existing records.
- Measure read latency, event-loop responsiveness, memory, and actual filesystem space reclaimed separately. SQLite free pages alone are not space returned to the OS; database reclamation must be explicitly coordinated with the work tracked in design(storage): decide where runtime.sqlite's one-time full VACUUM runs before converting it to incremental vacuum #6000.
The initial rollout should keep the compression writer disabled until correctness, migration, and performance criteria have been reviewed. Benchmark on disposable copies rather than user databases.
Alternatives or workarounds
Manual history cleanup, keeping all content as plain JSON, and compressing a whole Session archive. Per-record compression is a candidate because it can retain indexed access and append new content without rewriting the complete old transcript. Retention and cleanup remain useful independent controls.
AI-assisted issue prepared with OpenAI Codex at @testikun's request.
Activity
testikun commentedon Oct 15, 2026
take
testikun commentedon Oct 19, 2026
@liugddx could we get your guidance on the existing-database boundary for this cold-storage work, in light of your latest direction in #6000?
The connection between the two issues is that lossless compression frees pages inside SQLite, but an existing
auto_vacuum=NONEdatabase cannot return those pages to the filesystem incrementally. We explored that dependency in #6115. We have paused code changes to clarify the design direction.The current prototype opens the Host status endpoint first, then performs conversion before Ready and before opening ordinary business stores. It retains the Root owner, initializes the schema, closes the process-local operational database connection, and runs in-place
VACUUMplusintegrity_checkin a worker withsynchronous=FULL. Desktop displays the actual upgrade stage and elapsed time. The shared readiness helper givesupgrading_storagea separate 30-minute budget, while ordinary startup keeps its existing budget. The free-space precheck requires approximately 2x the current database file size plus 64 MiB in additional free space. The parent retains ownership until the worker exits.Compression itself is bounded background work after Ready, for completed large records after seven days without an actual local content write. Reads keep the stored representation compressed.
We measured the following on native macOS, Apple M2 Pro, 16 GiB RAM, Node 24.20.0, using a historical synthetic fixture with 185,930 RuntimeEvents / 883,500,249 bytes of original event JSON. All writes used disposable copies; the source remained unchanged. The disk reductions below measure compression followed by reclamation.
The complete historical file was 1,158,852,608 bytes before conversion alone. The replay measures a different database layout, rather than a full source-schema migration. These separate runs do not establish a before/after speedup or a dependable seconds-per-GB estimate. Sampled parent event-loop maximum gaps during the upgrades were approximately 9.7 ms and 29.7 ms; this does not qualify every startup stage.
Additional checks and costs:
The prototype still has unresolved boundaries that matter to this decision:
require_currentopeners has not been established by the Root/process-local checks.Would you consider controlled pre-Ready in-place conversion worth evaluating once those coordination and lifecycle gaps are addressed, or should we separate legacy conversion from this cold-compression work and pursue the
VACUUM INTOstaging/promotion investigation under #6000? In particular, we would appreciate your view on whether an extended upgrade startup budget is itself an unacceptable Desktop tradeoff. We want to settle that direction before resuming implementation changes.AI assistance: OpenAI Codex helped prepare the implementation, experiments and this comment at @testikun's request.
youlemei2022-web commentedon Oct 20, 2026
Thanks @testikun — the measurements and the caveats are both useful, and I think they already answer the first question.
1. Separate legacy conversion from this work. Keep #6115 to compression + bounded reclamation only: no automatic conversion, no schema commit that makes the file incompatible with older binaries. The conversion belongs to #6000, pursued as
VACUUM INTOstaging plus a promotion step at the next start, not as an in-place upgrade on the startup path.2. An extended upgrade budget is not an acceptable default. 64–86 s for 3 GiB is already past the 75 s deadline before
integrity_checkand before slower disks; a 30-minuteupgrading_storagewindow only converts "startup fails" into "app appears hung for minutes", which is worse for Desktop. If we ever support in-place conversion, it has to be explicit and user-visible: opt-in, a hard cap, real progress with elapsed time, and a documented fallback to the old binary. Not a silent pre-Ready step.3. What the staging design needs before it can be evaluated.
runtime.sqlite, not just Root/process-local — including the CLI managed-service verifier, whose 45 s deadline must not be able to kill a conversion worker.integrity_check+ expected size, then swap; a crash at any point leaves the original file usable and retryable.VACUUM INTOand the promotion step themselves, at least one run on native macOS, plus disk-full and mid-promotion crash cases.So: don't resume the pre-Ready conversion path in #6115. If you want to take the staging design, #6000 is the place, and the fencing + promotion-failure matrix is the first deliverable — I'd review that before any implementation.
fix(ui): point the model-picker story at the moved brand-mark module