feat(v0.12): Memory v2 — FTS5 recall, snapshot compaction, cross-project memory

CHANGELOG.md

@@ -6,6 +6,28 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.12.0] — 2026-07-05
+
+### Added
+- **FTS5 semantic recall** (schema v7): episodic recall now ranks by SQLite
+  FTS5 BM25 over goal/summary/tags when the SQLite build supports it, blended
+  with the existing tag/task_type boosts and recency tie-break. Falls back to
+  the deterministic keyword scorer when FTS5 is unavailable — no new dependency,
+  callers unchanged (the `store.search_episodes` signature is stable). The
+  `episodes_fts` index is built and self-healed by `LoopStore._ensure_fts5`,
+  covering the v6→v7 upgrade of stores with pre-existing episodes.
+- **Opt-in cross-project recall**: `loopllm_recall(scope="global")` (MCP) and
+  `loopllm recall <query> --global` (CLI) fan out over every project under
+  `~/.loopllm/projects/`, tagging each hit with its `project_id`. Per-project
+  isolation remains the default; this restores cross-repo institutional memory
+  as an explicit choice.
+
+### Changed
+- **Recovery-snapshot compaction**: `AgentLoopSession.to_snapshot()` compacts
+  intermediate `step_outputs` to head+tail stubs (keeping the final artifact
+  whole — the only one read back), bounding `active_runs` growth on long loops
+  with large artifacts. Recovery behaviour is unchanged.
+
 ## [0.11.0] — 2026-07-05
 
 ### Added

CONTRIBUTING.md

@@ -9,7 +9,7 @@ the branch/commit conventions, and the checks your change must pass.
 git clone https://github.com/azank1/loop-llm
 cd loop-llm
 pip install -e ".[dev]"
-python -m pytest tests/ -q          # 298 tests (294 pass, 4 integration skipped)
+python -m pytest tests/ -q          # 312 tests (308 pass, 4 integration skipped)
 ```
 
 ## Branch naming convention

README.md

@@ -21,7 +21,7 @@ training, nothing leaves your machine.
 renders live. A goal becomes nodes; each node is scored independently before
 its dependents unlock; a failed node shows why in plain language.*
 
-> Current release: **v0.11.0** — episodic memory (v0.8), DAG virtual sub-agents (v0.9), per-project scoping + verification audit trail (v0.10), and the CI verification gate (v0.11). See [CHANGELOG.md](CHANGELOG.md).
+> Current release: **v0.12.0** — episodic memory (v0.8), DAG virtual sub-agents (v0.9), per-project scoping + verification audit trail (v0.10), CI verification gate (v0.11), and FTS5 semantic recall + cross-project memory (v0.12). See [CHANGELOG.md](CHANGELOG.md).
 
 ---
 
@@ -43,14 +43,17 @@ PromptLoop uses two complementary memory layers in this per-project store:
 | Layer | What it learns | MCP tools |
 |---|---|---|
 | **Meta-memory** (v0.6+) | Optimal loop depth, convergence rate, scoring weights | `loopllm_loop_end`, `loopllm_feedback` |
-| **Episodic memory** (v0.8+) | Summaries of past loops/plans — keyword recall | `loopllm_recall`, `loopllm_run_status`, `loopllm_loop_resume` |
+| **Episodic memory** (v0.8+) | Summaries of past loops/plans — FTS5 relevance recall | `loopllm_recall`, `loopllm_run_status`, `loopllm_loop_resume` |
 
 Episodic memory is **not** full chat RAG — it stores compressed outcomes so the next
 loop of the same task type can recall *what worked before*. Recall is also injected
 automatically: `loopllm_loop_start` returns `similar_episodes`, and
-`loopllm_intercept` flags `recall_available` on clear prompts. (Ranking is
-deterministic keyword overlap today; the seam is stable for an FTS5/vector upgrade
-in v0.12.)
+`loopllm_intercept` flags `recall_available` on clear prompts. Ranking is SQLite
+FTS5 (BM25 over goal/summary/tags) blended with tag/task_type boosts and recency,
+falling back to deterministic keyword overlap when a SQLite build lacks FTS5.
+Recall is per-project by default; pass `scope="global"` to `loopllm_recall`
+(or `loopllm recall <query> --global`) to search across every project on the
+machine, each hit tagged with its project id.
 
 **Verification audit trail.** Every recorded episode (agent-loop, DAG node, or
 DAG merge) is stamped with the git commit that was `HEAD` at the time. Run

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "loopllm"
-version = "0.11.0"
+version = "0.12.0"
 description = "MCP server that observes every prompt, scores quality in real time, and closes the loop with iterative refinement. Built on FastMCP, SQLite, and Bayesian priors — no extra LLM required."
 readme = "README.md"
 requires-python = ">=3.11"

src/loopllm/__init__.py

@@ -29,7 +29,7 @@
 from loopllm.store import LoopStore, SQLiteBackedPriors
 from loopllm.tasks import Task, TaskOrchestrator, TaskPlan, TaskState
 
-__version__ = "0.11.0"
+__version__ = "0.12.0"
 
 __all__ = [
     # Engine

src/loopllm/agent_loop.py

@@ -7,6 +7,7 @@
 """
 from __future__ import annotations
 
+import hashlib
 import time
 import uuid
 from dataclasses import dataclass, field
@@ -28,6 +29,25 @@
 
 MAX_STEPS = MAX_STEPS_DEFAULT
 
+# Head/tail characters retained when compacting an intermediate step artifact
+# in a recovery snapshot. Only the final step output is ever read back (via
+# ``step_outputs[-1]``); earlier ones exist solely for the crash-recovery
+# snapshot, so truncating them caps ``active_runs`` growth on long loops with
+# large artifacts without losing anything a resume or ``loop_end`` consumes.
+_STEP_OUTPUT_KEEP = 500
+
+
+def _compact_step_output(text: str) -> str:
+    """Truncate a long intermediate step artifact to a head+tail stub."""
+    if len(text) <= 2 * _STEP_OUTPUT_KEEP + 80:
+        return text
+    sha = hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
+    return (
+        f"{text[:_STEP_OUTPUT_KEEP]}\n"
+        f"…[compacted {len(text)} chars, sha256:{sha}]…\n"
+        f"{text[-_STEP_OUTPUT_KEEP:]}"
+    )
+
 
 @dataclass
 class AgentLoopSession:
@@ -75,8 +95,19 @@ class AgentLoopSession:
     )
 
     def to_snapshot(self) -> dict[str, Any]:
-        """Serialise this session for the ``active_runs`` recovery store."""
-        return {name: getattr(self, name) for name in self._SNAPSHOT_FIELDS}
+        """Serialise this session for the ``active_runs`` recovery store.
+
+        Intermediate ``step_outputs`` are compacted to head+tail stubs; the
+        final one is kept whole (it's the only entry ever read back). This
+        bounds snapshot size on long loops without changing recovery behaviour.
+        """
+        snapshot = {name: getattr(self, name) for name in self._SNAPSHOT_FIELDS}
+        outputs = snapshot.get("step_outputs") or []
+        if len(outputs) > 1:
+            snapshot["step_outputs"] = [
+                _compact_step_output(o) for o in outputs[:-1]
+            ] + [outputs[-1]]
+        return snapshot
 
     @classmethod
     def from_snapshot(cls, state: dict[str, Any]) -> AgentLoopSession:

src/loopllm/cli.py

@@ -502,6 +502,43 @@ def cmd_audit(args: argparse.Namespace) -> None:
     store.close()
 
 
+def cmd_recall(args: argparse.Namespace) -> None:
+    """Recall past episodes relevant to a query — this project, or all projects.
+
+    Defaults to the current project's store. ``--global`` fans out across every
+    project under ``~/.loopllm/projects/`` and tags each hit with its project id
+    — the human-facing view of the same cross-project recall the MCP
+    ``loopllm_recall(scope="global")`` tool exposes to the agent.
+    """
+    from loopllm.episodes import EpisodicStore, global_recall
+
+    if args.all_projects:
+        base = resolve_db_path(args.db).parent.parent.parent
+        episodes = global_recall(
+            args.query, task_type=args.task_type, k=args.k, base=base
+        )
+    else:
+        store = _get_store(args.db)
+        episodes = EpisodicStore(store).recall(
+            args.query, task_type=args.task_type, k=args.k
+        )
+        store.close()
+
+    if args.json:
+        print(json.dumps(episodes, indent=2, default=str))
+        return
+
+    if not episodes:
+        print("No matching episodes.")
+        return
+
+    for ep in episodes:
+        score = ep.get("score_final")
+        score_str = f"{score:.2f}" if score is not None else "—"
+        proj = f"[{ep['project_id']}] " if ep.get("project_id") else ""
+        print(f"{proj}score={score_str}  {ep['episode_type']:10s} {ep['goal'][:70]}")
+
+
 def cmd_audit_gate(args: argparse.Namespace) -> None:
     """CI gate: check that code-touching commits in a range carry a verification record.
 
@@ -798,6 +835,21 @@ def build_parser() -> argparse.ArgumentParser:
     )
     p_audit.set_defaults(func=cmd_audit)
 
+    # --- recall ---
+    p_recall = subparsers.add_parser(
+        "recall",
+        help="Recall past episodes relevant to a query (this project, or --global)",
+    )
+    p_recall.add_argument("query", help="What to recall (matched over goal/summary/tags)")
+    p_recall.add_argument(
+        "--global", dest="all_projects", action="store_true",
+        help="Search across every project under ~/.loopllm/projects/, tagged by project",
+    )
+    p_recall.add_argument("--task-type", default=None, help="Filter by task_type")
+    p_recall.add_argument("-k", type=int, default=5, help="Max results (default: 5)")
+    p_recall.add_argument("--json", action="store_true", help="Output JSON")
+    p_recall.set_defaults(func=cmd_recall)
+
     # --- audit-gate ---
     p_gate = subparsers.add_parser(
         "audit-gate",

src/loopllm/episodes.py

@@ -1,10 +1,11 @@
 """Episodic memory: record and recall completed loop/plan outcomes.
 
-Recall ranking is deterministic weighted keyword overlap (see
+Recall ranking uses SQLite FTS5 (BM25 over goal+summary+tags) when the SQLite
+build supports it, blended with tag/task_type boosts and a recency tie-break;
+it falls back to deterministic keyword overlap otherwise (see
 ``store.search_episodes``). The public ``EpisodicStore.recall`` /
 ``store.search_episodes`` signature is stable, so the ranking backend can be
-upgraded to SQLite FTS5 (a virtual table over goal+summary+tags, no new deps)
-or an embedding index later without touching callers — planned for v0.12.
+swapped for an embedding index later without touching callers.
 """
 from __future__ import annotations
 
@@ -193,3 +194,36 @@ def run_status_snapshot(self) -> dict[str, Any]:
 def tokenize_for_recall(text: str) -> list[str]:
     """Split text into recall terms (stopword-filtered, deduped)."""
     return _recall_terms(text)
+
+
+def global_recall(
+    query: str,
+    *,
+    task_type: str | None = None,
+    k: int = 5,
+    base: Path | None = None,
+) -> list[dict[str, Any]]:
+    """Recall across *every* project's store under ``<base>/projects/``.
+
+    Per-project scoping (v0.10) keeps each repo's episodes isolated. This
+    read-only fan-out restores cross-project institutional memory as an
+    explicit opt-in: it queries each project's store, tags every hit with its
+    ``project_id``, and returns the ``k`` most recent matches overall (recency
+    is the only ordering comparable across independent FTS indexes).
+    """
+    from loopllm.project_scope import list_project_dirs
+
+    hits: list[dict[str, Any]] = []
+    for proj_dir in list_project_dirs(base):
+        db = proj_dir / "store.db"
+        if not db.exists():
+            continue
+        store = LoopStore(db_path=db)
+        try:
+            for ep in EpisodicStore(store).recall(query, task_type=task_type, k=k):
+                ep["project_id"] = proj_dir.name
+                hits.append(ep)
+        finally:
+            store.close()
+    hits.sort(key=lambda e: e.get("recorded_at") or "", reverse=True)
+    return hits[:k]

src/loopllm/mcp_server.py

@@ -1696,10 +1696,21 @@ def _tool_recall(
     query: str,
     task_type: str | None = None,
     k: int = 5,
+    scope: str = "project",
 ) -> str:
-    """Recall similar past episodes by keyword search."""
-    episodes = _get_episodic().recall(query, task_type=task_type, k=k)
-    return json.dumps({"query": query, "count": len(episodes), "episodes": episodes}, indent=2)
+    """Recall similar past episodes. ``scope="global"`` fans out across all projects."""
+    if scope == "global":
+        from loopllm.episodes import global_recall
+
+        db_path = resolve_db_path(os.environ.get("LOOPLLM_DB"))
+        base = db_path.parent.parent.parent  # <base>/projects/<id>/store.db -> <base>
+        episodes = global_recall(query, task_type=task_type, k=k, base=base)
+    else:
+        episodes = _get_episodic().recall(query, task_type=task_type, k=k)
+    return json.dumps(
+        {"query": query, "scope": scope, "count": len(episodes), "episodes": episodes},
+        indent=2,
+    )
 
 
 def _tool_run_status() -> str:
@@ -2927,16 +2938,19 @@ def loop_status(session_id: str) -> str:
     @mcp.tool(
         name="loopllm_recall",
         description=(
-            "Recall similar past episodes (completed loops/plans) by keyword "
-            "search over goal, summary, and tags in ~/.loopllm/store.db."
+            "Recall similar past episodes (completed loops/plans) ranked by "
+            "relevance over goal, summary, and tags. Defaults to this project's "
+            "store; pass scope='global' to search across every project on this "
+            "machine (each hit tagged with its project_id)."
         ),
     )
     def recall(
         query: str,
         task_type: str | None = None,
         k: int = 5,
+        scope: str = "project",
     ) -> str:
-        return _tool_recall(query, task_type, k)
+        return _tool_recall(query, task_type, k, scope)
 
     @mcp.tool(
         name="loopllm_run_status",

src/loopllm/project_scope.py

@@ -84,6 +84,18 @@ def project_state_dir(base: Path | None = None, cwd: Path | None = None) -> Path
     return base / "projects" / resolve_project_id(cwd)
 
 
+def list_project_dirs(base: Path | None = None) -> list[Path]:
+    """Return every per-project state directory under ``<base>/projects/``.
+
+    Used by opt-in cross-project recall to fan out over all of a machine's
+    projects. Returns an empty list when no projects directory exists.
+    """
+    projects = (base or (Path.home() / ".loopllm")) / "projects"
+    if not projects.is_dir():
+        return []
+    return sorted(p for p in projects.iterdir() if p.is_dir())
+
+
 def current_commit_sha(cwd: Path | None = None) -> str | None:
     """Return the current ``HEAD`` commit sha, or None outside a git repo.