Skip to content

Checkpoint semantic cache per chunk so interrupted runs resume#1715

Open
A-Levin wants to merge 1 commit into
Graphify-Labs:v8from
A-Levin:feat/incremental-semantic-cache
Open

Checkpoint semantic cache per chunk so interrupted runs resume#1715
A-Levin wants to merge 1 commit into
Graphify-Labs:v8from
A-Levin:feat/incremental-semantic-cache

Conversation

@A-Levin

@A-Levin A-Levin commented Jul 7, 2026

Copy link
Copy Markdown

Problem

Semantic extraction only writes to the cache once, at the very end of the runsave_semantic_cache is called in __main__ after extract_corpus_parallel returns. on_chunk_done only prints progress; nothing is persisted per chunk.

So a run interrupted partway — a crash, a kill, a reboot, or a claude-cli/API run that exits when it hits a rate limit — loses every completed chunk and restarts from scratch. On a large corpus (thousands of files) with a slow local backend (e.g. Ollama), that can throw away many hours of completed work, and there is no way to make incremental progress across sessions.

Fix

Persist each chunk's semantic results to the cache as soon as the chunk completes, in both the serial and threaded paths of extract_corpus_parallel. On restart, check_semantic_cache already filters out cached files, so the run resumes where it left off.

To stay correct when a large file is split into slices across several chunks, save_semantic_cache gains a merge_existing flag: the checkpoint unions with any existing cache entry for a file instead of letting a later chunk overwrite an earlier slice.

  • Best-effort: a cache write error never aborts extraction.
  • Opt-out via GRAPHIFY_NO_INCREMENTAL_CACHE.
  • Backward compatible: save_semantic_cache defaults to merge_existing=False (unchanged behaviour); the final end-of-run save is untouched.

Testing

  • Existing suite green: test_cache.py (31), plus semantic tests (test_semantic_cleanup, test_zero_node_no_cache, test_word_count_cache, test_semantic_id_remap_root) — 35 more pass.
  • Added functional check: two slices of the same file written with merge_existing=True accumulate (n1n1,n2); merge_existing=False still overwrites; check_semantic_cache then reports the file as cached (resume works).
  • Verified live on a ~5000-file vault with the Ollama backend: cache entries now appear after the first chunk instead of staying empty until the run finishes.

Semantic extraction only wrote to the cache once, at the very end of the
run (save_semantic_cache in __main__ after extract_corpus_parallel returns).
A run interrupted partway — a crash, a kill, or a claude-cli/API run that
exits when it hits a rate limit — therefore lost every completed chunk and
restarted from scratch. On a large corpus with a slow local backend this can
throw away many hours of work.

Persist each chunk's results to the semantic cache as soon as it completes,
in both the serial and threaded paths of extract_corpus_parallel. Add a
merge_existing option to save_semantic_cache so a file split into slices
across several chunks accumulates its slices instead of the later chunk
overwriting the earlier one. The checkpoint is best-effort (a cache write
error never aborts extraction) and can be disabled with
GRAPHIFY_NO_INCREMENTAL_CACHE. Default behaviour of save_semantic_cache is
unchanged (merge_existing defaults to False).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant