Skip to content

feat(tracking): link hook invocations and measure recall output - #4541

Open
yeahjack wants to merge 12 commits into
rtk-ai:developfrom
yeahjack:feat/recall-invocation-accounting
Open

yeahjack wants to merge 12 commits into
rtk-ai:developfrom
yeahjack:feat/recall-invocation-accounting

Conversation

@yeahjack

@yeahjack yeahjack commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add explicit local associations and measure the output cost of actual recall executions, rather than treating deduplicated content reads as execution counts.

  • Assign an opaque process invocation ID; link supported native Claude hook rewrites through a validated, non-global --hook-event-id option. Old history and unsupported shell shapes keep nullable associations, with no timestamp/command-text inference.
  • Add immutable capture IDs independent of content hashes, capped at 1,000 metadata records and pruned with payload eviction. Legacy hash recalls deliberately remain source-unassigned, including after older metadata is pruned. Capture-addressed hints can build on this seam separately.
  • Record each actual recall emission's stdout bytes and estimated tokens, including repeated reads, selection/grep results, empty matches and bytes written before an output error. Existing unique-content recall counters retain their meaning.
  • Export additive output_accounting, plus up to 100 command and recall-event details with rtk gain --history --format json. Signed observed net output savings deduct initial output and measured retrieval output. Coverage/unknown counts and measurement_complete: false prevent conflating this with complete context, billing or task-success measurements.
  • Add atomic, additive tracking migrations; existing 90-day history retention and reset include recall events. IDs are not included in telemetry.

Accounting is by each event's date and working directory: recalling an older capture charges the current retrieval event. External tee reads, stderr, historical reads and failed tracking are not measured. Native Claude simple rewrites are the initial propagation scope; compound commands/wrappers and other integrations preserve their current shell/permission behavior and remain unassociated.

Motivation

This does not implement the broader agent-label grouping proposed by #1314, nor claim to replace end-to-end benchmark instrumentation.

Verification

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --offline
  • cargo test --all --offline: 4,388 passed, 8 existing ignored tests
  • Regression coverage: explicit joins for identical repeated commands; legacy migration/idempotence; signed net output savings; repeated/partial/grep/empty recalls; binary/partial stdout writes; immutable capture IDs and bounded metadata; project scoping, retention and reset; real hook-to-command-to-JSON export; quoted/compound/env-wrapper and nested-option behavior.

Release spot-checks on Linux compared develop 311ebcb with this PR at 85fe79b (60 measured runs after 5 warmups per case):

  • rtk git log --oneline -1: median 4.834 ms → 4.561 ms
  • Native Claude hook with session/tool metadata: median 13.624 ms → 14.016 ms (+0.392 ms)
  • rtk recall abc123 --from 901 on a 1,000-line stored fixture: median 2.736 ms → 2.635 ms

These shared-host spot-checks are regression evidence, not speedup claims; the baseline hook already exceeds 10 ms on this host. A real 250-file listing also verified that capture and command invocation IDs agree, and two 40-byte partial recalls charged 20 estimated tokens in total.

Cross-platform CI passed on commit 85fe79b282bd41097ae2f8b4ac93d63011acf194, including Linux, macOS, Windows, benchmark, clippy, formatting, rustdoc and security jobs.

@rtk-wshm-sync-bot

Copy link
Copy Markdown

wshm · Automated triage by AI

📊 Automated PR Analysis

✨ Type feature
🟡 Risk medium

Summary

Adds tracking infrastructure to link native Claude hook invocations to specific commands via a validated --hook-event-id flag, and measures actual output bytes/tokens from recall executions rather than inferring cost from deduplicated content reads. Introduces immutable capture IDs with bounded, pruned metadata, an additive output_accounting export in rtk gain --history --format json, and corresponding atomic migrations and retention/reset handling.

Review Checklist

  • Tests present
  • Breaking change
  • Docs updated

Analyzed automatically by wshm · This is an automated analysis, not a human review.

@yeahjack
yeahjack marked this pull request as ready for review October 12, 2026 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant