Skip to content

feat(memory): Frozen snapshot + cache invalidation for inject_memories #3017

@hamza-jeddad

Description

@hamza-jeddad

Background

Sub-issue of #3011. To preserve LLM prefix-cache stability (a key requirement called out in the Hermes memory architecture), the inject_memories builtin maintains a frozen snapshot of injected memory text and rebuilds it only on well-defined events.

Motivation

Without snapshotting, every memory write or every minor DB change could invalidate the model's prefix cache mid-session, hurting cache hit rate and latency. Mirroring Hermes' two-tier approach gives us:

  • A frozen snapshot used to build the injected block on each turn
  • A live mutable state in the DB for reads/writes
  • Explicit invalidation on memory writes and on context compaction

Scope

Snapshot lifecycle

Session start
    │
    ▼
Load snapshot from DB (initial materialisation)
    │
    ▼  ← used by inject_memories on every turn
Memory write (add_memory / update_memory / delete_memory)
    │
    ▼ invalidate → rebuild on next turn
Context compaction event
    │
    ▼ invalidate → rebuild on next turn

Where it lives

  • Snapshot is in-process only — never persisted
  • Stored alongside the agent's runtime state (one snapshot per agent per session)
  • Thread-safe with RWMutex (reads on turn_start are hot path)

Implementation Checklist

  • Snapshot struct holding cached []database.UserMemory + generation counter
  • Invalidate() method that bumps the generation counter
  • Hook into the memory tool write paths (add_memory, update_memory, delete_memory) to call Invalidate()
  • Hook into the compaction event in pkg/runtime/compactor/ (or wherever the compaction signal is emitted) to call Invalidate()
  • inject_memories builtin reads from snapshot; on stale generation, refetches from DB and stores
  • Unit tests: snapshot returns same bytes across turns when nothing changed; invalidates correctly on each trigger
  • Benchmark / micro-test: turn_start with cached snapshot should not hit the DB

Acceptance Criteria

  • Two consecutive turns with no memory writes produce byte-identical injected blocks
  • add_memory invalidates the snapshot → next turn reflects the new memory
  • update_memory invalidates the snapshot → next turn reflects the change
  • delete_memory invalidates the snapshot → next turn no longer shows the deleted entry
  • Context compaction triggers a rebuild
  • Snapshot is dropped on session end (no cross-session leakage)
  • No data races detected by go test -race

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/agentFor work that has to do with the general agent loop/agentic features of the apparea/toolsFor features/issues/fixes related to the usage of built-in and MCP tools

    Type

    No fields configured for Task.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions