Skip to content

feat(memory): spoke-memory optional capability family — wire, operations, engine, Connect, conformance - #114

Merged
auto-wood merged 103 commits into
mainfrom
iteration/v0.15-memory
Oct 29, 2026
Merged

auto-wood merged 103 commits into
mainfrom
iteration/v0.15-memory

Conversation

@auto-wood

Copy link
Copy Markdown
Contributor

Summary

Lands the optional spoke-memory capability family — per-subject subjective memory (impressions, not canon) with candidate→review→retain gating, explicit-only forgetting, payload-bound idempotency, single-authority spaces, and cooperative-provider transfer — as a complete wire-to-conformance slice.

  • 79 new wire schemas (13 data models + 33 memory.* op pairs; inventory 34 → 113) + regenerated TS/Rust types
  • MemoryPort (Rust trait + TS interface, standalone like ExtractionPort) + pure validators, grant-coverage, JCS digest helpers, 14 reject codes
  • crates/spoke-memory reference engine: classify / recall / reflect / transfer — deterministic, I/O-free, RFC 8785-canonical (ES6 number semantics via ryu)
  • Connect integration: 33 dispatch rows under spoke-memory / spoke-memory-reflection / spoke-memory-transfer (Rust + TS), RemoteMemoryService + typed MemoryPortService delegation, client adapters, golden vectors
  • Conformance: 78-case corpus (44 wire) proven over the Connect loopback against TWO providers — SQLite reference provider 44/44 pass; loopback host 16 pass + 28 documented capability-gap inconclusives
  • Normative spec (MEM-R001–R027) + capability ADR + Author-Memory boundary amendments + integrator docs (EN/zh) + release-readiness verification (no version bump in this PR)

Protocol source: nexus research package 01-spoke-memory-protocol.md (proposed), landed with repo-side adjustments recorded in the iteration compass (D8 chunk-header extensions carrier; semantic locks: no affect wire field, explicit-only forgetting, flat subjects).

Residual disclosure (open issues, none blocker-deferred)

id severity tracking note
I-000021 medium store, linked p2/p3 verify-codegen does not compile downstream workspace crates (repairs landed in-iteration; gate fix is follow-up)
I-000023 medium store, linked p3/p5 content byte-measure divergence between spoke-operations helper and engine (documented, unification pending)
I-000025 medium store, linked p6 release.yml publish-crates omits crates/spoke-memory (separate authorized release-pipeline work)
I-000026 medium store, linked p6 CI has no test step for crates/spoke-memory
I-000019 low store, linked p1 large generated Rust files (compile/tooling cost)
I-000020 nit store, linked p1 receipts-get TS anonymous OneOfN branches
I-000022 nit store, linked p2 memory helper hot-path allocations
I-000024 nit store, linked p3 engine hot-path allocation advisories
I-000027 nit store, linked p5 conformance harness hardening advisories

Merge policy

pr_merge_policy: reviewed-auto — this PR is expected to receive a merge approval via the PR-level review loop (ship it on the current head), not a human owner gate.

Verification

  • pnpm run verify-codegen exit 0 (113 schemas; opaque-JSON + hoist asserts)
  • Whole connect suite 371+ green (dispatch/loopback/golden/conformance/sqlite_provider)
  • cargo +nightly check --workspace clean; pnpm run docs:build + pnpm run verify:version green
  • Plan-level QC tri + QA gates all recorded per plan (6/6 Done)

The pending candidate's `proposed` (kind + content) and the consumed shell
(kind only) shared the JSON title "Proposed", so codegen merged them into one
kind-only type and a legal pending candidate with `proposed.content` could not
deserialize. The consumed shell is now `ConsumedProposed`; regenerated TS and
Rust outputs follow, and a regression test covers the pending shape, the
consumed shell, and the rejection of content on a consumed candidate.

Also repairs the pre-existing `builder_smoke` assumption that
`Scope.extensions` is a bare map (P1 codegen made it `Option<ExtensionMap>`).
The regenerated Connect types replaced the per-variant extension key aliases
with the shared `ExtensionKey`, and payload/extension fields became
newtypes (`Shared0`). Mechanical adaptation only: wrap the extension maps and
request payload in their newtypes, convert the invoke payload back to
`serde_json::Value` before dispatch, and follow the same renames in the
loopback fixtures. No behaviour change.
@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Critical impact] The latest changes appear safe to merge; the remaining malformed-receipt checks are non-blocking.

Findings

  1. P2 Malformed replies look successful ▶
Fix with agent prompt
### Issue 1
packages/spoke-connect-ts/src/remote/remote-adapter.ts:752-755
The new memory methods call `#invokeMapped`, but none of their operations appear in `SUCCESS_SHAPES`. `isValidSuccessPayload` therefore accepts any value, including `null` or a string, and returns it as a typed memory response. Callers can then fail when reading expected fields. Add required-field checks for memory reads and receipt states, matching the existing port checks.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds the optional spoke-memory family across schemas, operations, the Rust engine, Connect, and conformance tests.

  • Memory callers can capture candidates, review them, and work with retained records.
  • The reference engine classifies candidates and prepares budgeted memory recall.
  • Reflection requests can pin their inputs and report when those inputs change.
  • Memory providers can exchange verified, staged bundles between spaces.
  • Connect clients and hosts can invoke memory operations through negotiated capabilities.
  • Memory schemas and shared protocol documentation now feed generated client and Rust types.

Diagram

sequenceDiagram
  participant Client
  participant Connect
  participant Provider
  Client->>Connect: memory operation
  Connect->>Connect: Check negotiated capability
  Connect->>Provider: Invoke with authenticated peer
  alt Provider returns
    Provider-->>Connect: Result or receipt
    Connect-->>Client: Signed response
  else Response deadline expires
    Connect-->>Client: OUTCOME_UNCERTAIN
    Provider->>Provider: Continue admitted work
  else Provider task fails
    Connect-->>Client: OUTCOME_UNCERTAIN
  end
  Client->>Connect: Replay same key or read receipt
Loading

Reviews (3) · Last reviewed commit: "build(connect): refresh xcframework from..." · Reviewed by Greptile

Comment thread crates/spoke-memory/src/transfer.rs
Comment thread crates/spoke-connect/src/remote/responder.rs Outdated
Comment thread crates/spoke-connect/tests/support/sqlite_memory.rs Outdated
Comment thread crates/spoke-connect/tests/support/sqlite_memory.rs Outdated
Comment thread packages/spoke-connect-ts/src/remote/remote-adapter.ts
Comment thread crates/spoke-connect/tests/memory_conformance.rs Fixed
Comment thread crates/spoke-connect/tests/memory_loopback.rs Fixed
…est check

A chunker that drops an empty header.extensions (D8: omitted == empty
map) reassembled into a bundle without the key, so the sealed digest
computed over extensions: {} never verified. Normalize the omission
before hashing and cover the foreign-chunker shape with a regression
test.
…erve budget

The memory serve-budget timeout wrapped service.invoke in
tokio::time::timeout, whose expiry dropped the service future —
cancelling admitted provider work (the receipt write could be lost)
while answering OUTCOME_UNCERTAIN. The TS path leaves the provider
promise running; match it by spawning the invoke and racing the
deadline against a oneshot: expiry answers the budget, the detached
task runs to its real completion, and polling observes it. The
response vocabulary is unchanged (result if ready, else
OUTCOME_UNCERTAIN + retry_class uncertain); zero-budget and non-memory
timeouts are untouched.

RemoteServePorts::as_memory now returns an owned Arc so the spawned
task can own the face. The loopback suite gains a provider fixture
whose commit is part of the invoke future (no self-detach), proving
the commit lands after the deadline.
…keep get_receipt read-only

Two reviewed findings on the SQLite reference provider:

- recall: the cursor offset was applied AFTER engine_recall limited the
  page to max_items, so page 2+ was empty forever and the cursor never
  advanced. The engine RecallRequest now carries the offset (MEM-R026)
  and drops the leading ranked records before the item/byte budget; the
  provider passes it through. New walk-every-page test plus an
  engine-level offset test.
- receipts.get: polling ran complete_pending through mutate_value, a
  write on the read path (MEM-R011). The provider worker now runs
  separately (run_worker); the port read only observes stored state.
  The conformance interpreter drives the worker explicitly (honoring
  the corpus hold/release vocabulary), direct poll sites settle via
  run_worker, and a new test asserts polling alone leaves a pending
  receipt pending.
Memory ops had no entry in SUCCESS_SHAPES, so invokeMapped's shape gate
accepted any payload for every memory.* response (spokeOk(garbage)).
Add required-field checks mirroring the existing port checks and the
schemas/ops/memory-*-response success branches: read ops get their
declared top-level fields (describe/spaces/subjects/candidates/grants/
records/recall/profile/reflections/exports), and every mutating op plus
memory.receipts.get gets the receipt-envelope fields shared by the
committed/pending/failed states. FieldKind gains boolean for recall.truncated
and reflections.state.stale.
…ld comparisons) into iteration/v0.15-memory
@auto-wood

Copy link
Copy Markdown
Contributor Author

@greptile-apps review

Comment thread crates/spoke-connect/src/remote/responder.rs Outdated
…nonces

§14.3: an admitted memory invoke that dies after invocation (Rust: the
detached task panics and drops its result channel; TS: the provider
throws/rejects) left the responder answering a clean INTERNAL_ERROR,
which asserts no-effect the transport cannot support. Both paths now
answer OUTCOME_UNCERTAIN with details.retry_class = "uncertain" and
kind = "panic"; the Rust helper sits beside memory_uncertain_reject and
the TS memory catch matches it. INTERNAL_ERROR stays on paths where no
memory write is possible (extract, port, dispatch-denies).

Tests: a loopback fixture mode whose admitted capture panics proves the
Rust branch, and a TS responder test drives a throwing memory service.
Hello nonces in memory_loopback / memory_conformance are now derived
from the clock at runtime instead of hard-coded values (CodeQL
rust/hard-coded-cryptographic-value); the nonce is opaque to the
assertions.
@auto-wood

Copy link
Copy Markdown
Contributor Author

Independent review of head f0ebec9 — report at /tmp/pr-114-review.md (session-side; summary below).

All six Greptile findings are verified fixed at head with regression tests (transfer extensions normalization, responder deadline/panic → OUTCOME_UNCERTAIN with single-commit replay, recall cursor offset before budget, pure receipt reads, full SUCCESS_SHAPES coverage); local cargo (spoke-memory / spoke-operations / spoke-connect) and vitest (397 tests) pass at head, CI is green, and no code-scanning alerts remain open. One new blocking issue: the generated TS types keep idempotency_key optional on committed-receipt branches for 18 keyed mutation responses (the tightening allOf declares required: ["op","idempotency_key"] but the emitted interface only pins op), so TS consumers can build responses the schema and the Rust types reject — the remote adapter's runtime validation mitigates it at the Connect boundary, but the type drift needs a codegen/schema fix + regeneration before merge. Non-blocking P3s: an expect in recall that can panic on programmatically constructed non-finite floats, no non-BMP cross-vector between the two JCS implementations, and minor asymmetries in partition/reassemble and the responder's dead defensive branch. Everything else — determinism, digest math, idempotency, reads-never-write — reviewed sound.

Verdict: request changes (fix the committed-receipt idempotency_key TS drift; P3s optional).

… output

json-schema-to-typescript drops required-ness for names with no declared
property in the same schema: the op-pin allOf fragment required
[op, idempotency_key] but declared only op, so TS consumers could build
committed-receipt envelopes the schema rejects (Rust/typify kept the key
non-Option all along). Declare the idempotency_key property on the pin
fragment in the 18 keyed mutation response schemas (plus the same pin
shape in memory.receipts.get's failed branches): the emitted intersection
now carries idempotency_key as REQUIRED while Rust output stays
semantically unchanged (doc lines only). Regression: a compile-level
assertion in spoke-connect-ts (tsc fails if the key goes optional again)
plus a runtime round-trip — see tests/schemas/receipt-requiredness.test.ts.
- recall: the §9 byte measurement no longer panics on an unserializable
  content value — it returns the typed RecallError::NonFiniteContent,
  mirroring the non-finite guard in spoke-operations digests (serde_json
  Values cannot hold non-finite floats by construction, so the guard is
  defensive parity; documented at the site). recall() is now fallible;
  providers map the error to INVALID_INPUT and tests unwrap via a helper.
- partition_bundle now rejects a bundle whose header lacks extensions
  (D8): partitioning such a bundle would emit chunks whose reassembly
  normalizes the omission to {} and fails the bundle-digest check against
  the source digest. Test added; the corpus asset already carries
  extensions.
@auto-wood

Copy link
Copy Markdown
Contributor Author

Reviewer round fixes pushed (29a5f7d): the blocking P2 (TS requiredness for idempotency_key on the 18 keyed receipt branches — pin-fragment declaration repair, compile-level regression added) plus P3-1 (recall serialization guard) and P3-3 (partition asymmetry reject). P3-2/4/5 recorded as hardening advisories. Requesting re-review at 29a5f7d.

@auto-wood

Copy link
Copy Markdown
Contributor Author

@greptile-apps review

@auto-wood

Copy link
Copy Markdown
Contributor Author

Round-2 re-review of head 4c5c21d (round-1 report: /tmp/pr-114-review-r2.md, session-side; summary below).

The round-1 blocking P2 is properly fixed at the schema/codegen level: the keyed receipt allOf fragments now declare idempotency_key themselves, the generated TS emits it as REQUIRED across all keyed branches (33 required declarations confirmed; memory.subjects.ensure correctly stays optional per §6.10), and the new compile-level regression test (receipt-requiredness.test.ts) makes generator regressions a tsc failure. P3-1 landed as Result<RecallPage, RecallError::NonFiniteContent> with all call sites updated, and P3-3 landed as extensions in REQUIRED_HEADER_FIELDS with a partition-symmetry regression test. The xcframework now matches this head and CI is fully green. Local verification at 4c5c21d: cargo (spoke-memory 46 tests; spoke-connect + spoke-operations incl. the TS/Rust reject-code cross-check) and the vitest requiredness test all pass. Remaining round-1 P3s (non-BMP JCS cross-vector, minor nits) were declared optional and stay non-blocking. No new issues in the 3-commit increment.

Verdict: ship it

@auto-wood
auto-wood merged commit a6d9d9f into main Oct 29, 2026
31 of 32 checks passed
@auto-wood
auto-wood deleted the iteration/v0.15-memory branch October 29, 2026 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants