Checkpoint semantic cache per chunk so interrupted runs resume#1715
Open
A-Levin wants to merge 1 commit into
Open
Checkpoint semantic cache per chunk so interrupted runs resume#1715A-Levin wants to merge 1 commit into
A-Levin wants to merge 1 commit into
Conversation
Semantic extraction only wrote to the cache once, at the very end of the run (save_semantic_cache in __main__ after extract_corpus_parallel returns). A run interrupted partway — a crash, a kill, or a claude-cli/API run that exits when it hits a rate limit — therefore lost every completed chunk and restarted from scratch. On a large corpus with a slow local backend this can throw away many hours of work. Persist each chunk's results to the semantic cache as soon as it completes, in both the serial and threaded paths of extract_corpus_parallel. Add a merge_existing option to save_semantic_cache so a file split into slices across several chunks accumulates its slices instead of the later chunk overwriting the earlier one. The checkpoint is best-effort (a cache write error never aborts extraction) and can be disabled with GRAPHIFY_NO_INCREMENTAL_CACHE. Default behaviour of save_semantic_cache is unchanged (merge_existing defaults to False).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Semantic extraction only writes to the cache once, at the very end of the run —
save_semantic_cacheis called in__main__afterextract_corpus_parallelreturns.on_chunk_doneonly prints progress; nothing is persisted per chunk.So a run interrupted partway — a crash, a
kill, a reboot, or aclaude-cli/API run that exits when it hits a rate limit — loses every completed chunk and restarts from scratch. On a large corpus (thousands of files) with a slow local backend (e.g. Ollama), that can throw away many hours of completed work, and there is no way to make incremental progress across sessions.Fix
Persist each chunk's semantic results to the cache as soon as the chunk completes, in both the serial and threaded paths of
extract_corpus_parallel. On restart,check_semantic_cachealready filters out cached files, so the run resumes where it left off.To stay correct when a large file is split into slices across several chunks,
save_semantic_cachegains amerge_existingflag: the checkpoint unions with any existing cache entry for a file instead of letting a later chunk overwrite an earlier slice.GRAPHIFY_NO_INCREMENTAL_CACHE.save_semantic_cachedefaults tomerge_existing=False(unchanged behaviour); the final end-of-run save is untouched.Testing
test_cache.py(31), plus semantic tests (test_semantic_cleanup,test_zero_node_no_cache,test_word_count_cache,test_semantic_id_remap_root) — 35 more pass.merge_existing=Trueaccumulate (n1→n1,n2);merge_existing=Falsestill overwrites;check_semantic_cachethen reports the file as cached (resume works).