Skip to content

feat(v0.12): Memory v2 — FTS5 recall, snapshot compaction, cross-project memory - #12

Closed
azank1 wants to merge 4 commits into
az/ft/release-reconcile-v0.11from
az/ft/memory-v2
Closed

azank1 wants to merge 4 commits into
az/ft/release-reconcile-v0.11from
az/ft/memory-v2

Conversation

@azank1

@azank1 azank1 commented Jul 5, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #11

Built on az/ft/release-reconcile-v0.11 (#11, the v0.11.0 metadata reconcile). Until #11 merges, this PR's diff also shows #11's one commit — standard stacked-PR behavior; it narrows once #11 merges. Recommend merging #11 first, then GitHub auto-retargets this to main.

Summary

Cashes in the long-deferred episodic-recall upgrade seam and hardens the memory layer. Three independent pieces, each behind the existing stable interfaces (no caller changes):

1. FTS5 semantic recall (schema v7)

  • store.search_episodes now ranks by SQLite FTS5 BM25 over goal/summary/tags, blended with the existing tag/task_type boosts + recency tie-break — a strict improvement, not a reset. Signature unchanged; EpisodicStore.recall and the MCP path are untouched.
  • episodes_fts is an external-content virtual table kept in sync by triggers, built and self-healed by LoopStore._ensure_fts5 (idempotent on every open). The rebuild keys on table-newly-created rather than a row count, because an external-content table's COUNT(*) proxies episodes — so it correctly backfills the v6→v7 upgrade of a store that already has episodes.
  • Runtime fallback: _probe_fts5 detects availability once at init; recall degrades to the deterministic keyword scorer when FTS5 is absent or a MATCH fails. No new dependency.

2. Recovery-snapshot compaction

  • AgentLoopSession.to_snapshot() compacts intermediate step_outputs to head+tail stubs (sha + length), keeping the final artifact whole — the only entry ever read back (step_outputs[-1]). Bounds active_runs growth on long loops with large artifacts; recovery behavior unchanged.

3. Opt-in cross-project recall

  • Per-project scoping (v0.10) isolates repos; this restores cross-repo memory as an explicit choice. loopllm_recall(scope="global") (MCP) and loopllm recall <query> --global (CLI) fan out over ~/.loopllm/projects/*, tagging each hit with its project_id. Default stays per-project.

Test plan

  • ruff check src/ tests/ clean
  • mypy --strict src/loopllm/ — no issues in 28 files
  • pytest tests/ -q — 308 passed, 4 skipped (was 294)
  • python -m build — loopllm-0.12.0 wheel
  • Verified in a fresh clone + isolated venv (true CI-checkout conditions), all four gates green
  • New tests: tests/test_recall_fts.py (relevance, goal+summary match, FTS/keyword top-hit parity, reopen backfill, real v6→v7 migration recall, task_type filter), tests/test_global_recall.py (fan-out + project tagging, per-project isolation, k cap, CLI --global), plus compaction round-trip tests in tests/test_agent_loop.py.

Note (out of scope, flagging)

While adding recall --json I found the package configures no structlog handler, so store_schema_created (debug) prints to stdout on first-ever store creation — latent risk for --json piping and the MCP stdio stream. Pre-existing (affects merged audit --json/paths + the MCP server), so not fixed here. Happy to route structlog to stderr as a small follow-up if you want.

Publishing v0.12.0 (tag, release, PyPI, .vsix) remains a manual step per RELEASING.md.


Generated by Claude Code


Note

Medium Risk
Touches SQLite schema/migrations and recall ranking for all episodic memory; snapshot compaction affects persisted recovery state but is designed to preserve scores and the final artifact.

Overview
v0.12.0 upgrades episodic memory and tightens crash-recovery storage without changing public recall APIs.

FTS5 recall (schema v7): LoopStore.search_episodes now prefers SQLite FTS5 BM25 over goal/summary/tags, with the same tag/task_type boosts and recency tie-break as before; it falls back to the keyword scorer when FTS5 is missing or MATCH fails. An external-content episodes_fts index is created and backfilled via _ensure_fts5 on store open (including v6→v7 upgrades).

Snapshot compaction: AgentLoopSession.to_snapshot() stubs long intermediate step_outputs (head+tail + hash) while keeping the final artifact intact, limiting active_runs growth on long CDV loops.

Cross-project recall (opt-in): global_recall fans out over ~/.loopllm/projects/*, exposed as loopllm_recall(scope="global") and new CLI loopllm recall <query> --global, tagging hits with project_id. Per-project recall remains the default.

Reviewed by Cursor Bugbot for commit b0a6f76. Bugbot is set up for automated code reviews on this repo. Configure here.

azank1 added 4 commits July 5, 2026 20:10
Recall was deterministic keyword overlap; upgrade it to SQLite FTS5 (schema v7)
when the SQLite build supports it, blended with the existing tag/task_type
boosts and recency tie-break so results are a strict improvement, not a reset.

- episodes_fts external-content virtual table over goal/summary/tags, kept in
  sync via AFTER INSERT/UPDATE/DELETE triggers. Built and backfilled by
  LoopStore._ensure_fts5 (idempotent, self-healing on reopen) rather than a
  hard migration script, since FTS5 is a compile-time SQLite option.
- _probe_fts5 detects availability once at init; search_episodes falls back to
  the keyword scorer when FTS5 is absent or a MATCH fails. Callers unchanged.
- Rebuild keys on table-newly-created (external-content COUNT proxies episodes,
  so it can't signal an empty index) — covers the v6->v7-with-existing-episodes
  upgrade path.
- Tests: FTS relevance, goal+summary match, FTS/keyword top-hit parity, reopen
  backfill, real v6->v7 migration recall, task_type filter.
AgentLoopSession.to_snapshot() stored every step_output whole on every
checkpoint, so active_runs grew linearly with loop length x artifact size.
Only the final step output is ever read back (step_outputs[-1], via
summarize_artifacts and loop_end); intermediates exist solely for the
recovery snapshot.

Compact all-but-last step outputs to a head+tail stub (with a sha256 marker
and original length) when snapshotting; keep the final artifact verbatim. The
in-memory session is untouched, and restore is unaffected since only [-1] is
consumed. Short outputs pass through unchanged.

Tests: intermediate compaction + final preserved, snapshot->restore keeps
scores and final artifact and the loop still advances, short outputs verbatim.
Per-project scoping (v0.10) isolates each repo's episodes, which also removed
any way to reuse what worked in another project. Restore it as an explicit,
read-only opt-in:

- project_scope.list_project_dirs() enumerates ~/.loopllm/projects/*.
- episodes.global_recall() fans out over every project's store, tags each hit
  with its project_id, and returns the k most recent matches (recency is the
  only ordering comparable across independent FTS indexes).
- MCP loopllm_recall gains scope='project'|'global'; default unchanged.
- CLI: new `loopllm recall <query> [--global] [--task-type] [-k] [--json]`.

Tests: fan-out spans + tags projects, per-project isolation by default, k cap,
empty-when-no-projects, and the --global CLI path.
…ect memory

Version 0.11.0 -> 0.12.0; CHANGELOG [0.12.0] entry; README memory section
updated (FTS5 relevance recall + scope=global); CONTRIBUTING test count -> 308.
Repository owner deleted a comment from cursor Bot Aug 1, 2026
@azank1
azank1 deleted the branch az/ft/release-reconcile-v0.11 August 1, 2026 00:10
@azank1 azank1 closed this Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant