Skip to content

bug(storage): context-offload page reclamation never retries an unfinished WAL checkpoint #6149

Closed

Description

@liugddx

What happens

reclaimFreePages in packages/storage/src/sqlite-context-offload-store.ts (added in #5855) runs PRAGMA wal_checkpoint(PASSIVE) only in the call where incremental_vacuum reclaimed pages:

if (this.#readTransaction(() => this.#freelistPages()) === 0) {
  return { reclaimedPages: 0, hasMore: false };
}
// ...
if (result.reclaimedPages > 0) this.#database.exec('PRAGMA wal_checkpoint(PASSIVE)');

PASSIVE does not wait for readers. If another connection holds an older read snapshot while the last batch commits, the checkpoint stops early and the truncated pages stay in the WAL. The freelist is already empty at that point, so every later maintenance tick returns at the first check and never retries the checkpoint. The pages are reclaimed logically, but context-offload.sqlite does not shrink on disk until some unrelated write happens to trigger an autocheckpoint. On an idle install, that may never happen.

@Astro-Han reproduced the same pattern with real SQLite in the review of #6083, which copies this logic for runtime.sqlite: after 536 pages were drained with a reader pinned, the file stayed at 3,334,144 bytes; one extra PASSIVE checkpoint brought it to 1,138,688 bytes.

This is not data corruption: reads, the freelist accounting and quick_check are all correct. The problem is only that the disk space is not returned.

Expected

Once the freelist drains, a checkpoint that could not finish is retried later without blocking. The maintenance lane still never takes the write lock on an idle tick, never waits on readers, and never runs a full VACUUM.

Suggested fix

Read the row that wal_checkpoint(PASSIVE) returns (busy, log, checkpointed). If busy = 1 or checkpointed < log, keep an in-memory "checkpoint pending" flag. On the next tick, if the freelist is empty but the flag is set, run PASSIVE again outside any write transaction, and clear the flag once it completes. Return hasMore: false so the lane stays on its idle cadence instead of spinning while a reader holds its snapshot.

Add a regression test that holds a read transaction across the final batch, releases it, and asserts that the file shrinks on the next idle call.

Ideally runtime.sqlite (#6083) and context-offload share one small helper, so the two lanes do not drift apart. See my review on #6083.

Refs #5855, #6083, #6000

Activity

  1. Dante-dan commented on Oct 18, 2026

    @Dante-dan
    Contributor

    Automated contribution plan, prepared with OpenAI Codex:

    The empty-freelist return currently skips the only PASSIVE checkpoint path, so a checkpoint interrupted by a pinned reader loses its retry opportunity. I plan to keep the checkpoint completion state in memory, read the busy/log/checkpointed result, and retry a pending PASSIVE checkpoint on later idle calls after the freelist drains. The retry stays outside write transactions and returns hasMore: false so readers cannot cause a busy maintenance loop.

    I will put the checkpoint-state logic in a small internal storage helper that the runtime lane can reuse, while limiting this change to context-offload: #6083 currently remains runtime-only. I will add the requested real-SQLite regression holding a read transaction across the final reclamation batch, then releasing it and checking that the next idle call shrinks the main file, alongside the existing idle-lock and bounded-batch coverage.

  2. Dante-dan commented on Oct 18, 2026

    @Dante-dan
    Contributor

    take

  3. liugddx commented on Oct 18, 2026

    @liugddx
    MemberAuthor

    Thanks @Dante-dan, assigned. The plan matches what I had in mind: keep the checkpoint state in memory, retry PASSIVE outside write transactions after the freelist drains, return hasMore: false, and add the pinned-reader regression. Keeping the helper reusable but scoped to context-offload is fine. #6083 can adopt it for runtime.sqlite once both land. Please say Fixes #6149 in the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions