Repository navigation
Conversation
Recall was deterministic keyword overlap; upgrade it to SQLite FTS5 (schema v7) when the SQLite build supports it, blended with the existing tag/task_type boosts and recency tie-break so results are a strict improvement, not a reset. - episodes_fts external-content virtual table over goal/summary/tags, kept in sync via AFTER INSERT/UPDATE/DELETE triggers. Built and backfilled by LoopStore._ensure_fts5 (idempotent, self-healing on reopen) rather than a hard migration script, since FTS5 is a compile-time SQLite option. - _probe_fts5 detects availability once at init; search_episodes falls back to the keyword scorer when FTS5 is absent or a MATCH fails. Callers unchanged. - Rebuild keys on table-newly-created (external-content COUNT proxies episodes, so it can't signal an empty index) — covers the v6->v7-with-existing-episodes upgrade path. - Tests: FTS relevance, goal+summary match, FTS/keyword top-hit parity, reopen backfill, real v6->v7 migration recall, task_type filter.
AgentLoopSession.to_snapshot() stored every step_output whole on every checkpoint, so active_runs grew linearly with loop length x artifact size. Only the final step output is ever read back (step_outputs[-1], via summarize_artifacts and loop_end); intermediates exist solely for the recovery snapshot. Compact all-but-last step outputs to a head+tail stub (with a sha256 marker and original length) when snapshotting; keep the final artifact verbatim. The in-memory session is untouched, and restore is unaffected since only [-1] is consumed. Short outputs pass through unchanged. Tests: intermediate compaction + final preserved, snapshot->restore keeps scores and final artifact and the loop still advances, short outputs verbatim.
Per-project scoping (v0.10) isolates each repo's episodes, which also removed any way to reuse what worked in another project. Restore it as an explicit, read-only opt-in: - project_scope.list_project_dirs() enumerates ~/.loopllm/projects/*. - episodes.global_recall() fans out over every project's store, tags each hit with its project_id, and returns the k most recent matches (recency is the only ordering comparable across independent FTS indexes). - MCP loopllm_recall gains scope='project'|'global'; default unchanged. - CLI: new `loopllm recall <query> [--global] [--task-type] [-k] [--json]`. Tests: fan-out spans + tags projects, per-project isolation by default, k cap, empty-when-no-projects, and the --global CLI path.
…ect memory Version 0.11.0 -> 0.12.0; CHANGELOG [0.12.0] entry; README memory section updated (FTS5 relevance recall + scope=global); CONTRIBUTING test count -> 308.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #11
Built on
az/ft/release-reconcile-v0.11(#11, the v0.11.0 metadata reconcile). Until #11 merges, this PR's diff also shows #11's one commit — standard stacked-PR behavior; it narrows once #11 merges. Recommend merging #11 first, then GitHub auto-retargets this tomain.Summary
Cashes in the long-deferred episodic-recall upgrade seam and hardens the memory layer. Three independent pieces, each behind the existing stable interfaces (no caller changes):
1. FTS5 semantic recall (schema v7)
store.search_episodesnow ranks by SQLite FTS5 BM25 over goal/summary/tags, blended with the existing tag/task_type boosts + recency tie-break — a strict improvement, not a reset. Signature unchanged;EpisodicStore.recalland the MCP path are untouched.episodes_ftsis an external-content virtual table kept in sync by triggers, built and self-healed byLoopStore._ensure_fts5(idempotent on every open). The rebuild keys on table-newly-created rather than a row count, because an external-content table'sCOUNT(*)proxiesepisodes— so it correctly backfills the v6→v7 upgrade of a store that already has episodes._probe_fts5detects availability once at init; recall degrades to the deterministic keyword scorer when FTS5 is absent or a MATCH fails. No new dependency.2. Recovery-snapshot compaction
AgentLoopSession.to_snapshot()compacts intermediatestep_outputsto head+tail stubs (sha + length), keeping the final artifact whole — the only entry ever read back (step_outputs[-1]). Boundsactive_runsgrowth on long loops with large artifacts; recovery behavior unchanged.3. Opt-in cross-project recall
loopllm_recall(scope="global")(MCP) andloopllm recall <query> --global(CLI) fan out over~/.loopllm/projects/*, tagging each hit with itsproject_id. Default stays per-project.Test plan
ruff check src/ tests/cleanmypy --strict src/loopllm/— no issues in 28 filespytest tests/ -q— 308 passed, 4 skipped (was 294)python -m build—loopllm-0.12.0wheeltests/test_recall_fts.py(relevance, goal+summary match, FTS/keyword top-hit parity, reopen backfill, real v6→v7 migration recall, task_type filter),tests/test_global_recall.py(fan-out + project tagging, per-project isolation, k cap, CLI--global), plus compaction round-trip tests intests/test_agent_loop.py.Note (out of scope, flagging)
While adding
recall --jsonI found the package configures no structlog handler, sostore_schema_created(debug) prints to stdout on first-ever store creation — latent risk for--jsonpiping and the MCP stdio stream. Pre-existing (affects mergedaudit --json/paths+ the MCP server), so not fixed here. Happy to route structlog to stderr as a small follow-up if you want.Publishing
v0.12.0(tag, release, PyPI,.vsix) remains a manual step perRELEASING.md.Generated by Claude Code
Note
Medium Risk
Touches SQLite schema/migrations and recall ranking for all episodic memory; snapshot compaction affects persisted recovery state but is designed to preserve scores and the final artifact.
Overview
v0.12.0 upgrades episodic memory and tightens crash-recovery storage without changing public recall APIs.
FTS5 recall (schema v7):
LoopStore.search_episodesnow prefers SQLite FTS5 BM25 over goal/summary/tags, with the same tag/task_typeboosts and recency tie-break as before; it falls back to the keyword scorer when FTS5 is missing or MATCH fails. An external-contentepisodes_ftsindex is created and backfilled via_ensure_fts5on store open (including v6→v7 upgrades).Snapshot compaction:
AgentLoopSession.to_snapshot()stubs long intermediatestep_outputs(head+tail + hash) while keeping the final artifact intact, limitingactive_runsgrowth on long CDV loops.Cross-project recall (opt-in):
global_recallfans out over~/.loopllm/projects/*, exposed asloopllm_recall(scope="global")and new CLIloopllm recall <query> --global, tagging hits withproject_id. Per-project recall remains the default.Reviewed by Cursor Bugbot for commit b0a6f76. Bugbot is set up for automated code reviews on this repo. Configure here.