Repository navigation
test: add agent fixtures for export→pytest integration tests - #25
Merged
utsab345 merged 1 commit intoOct 20, 2026
Merged
Conversation
Add two offline test agents (triage_agent, crash_agent) and a fake LLM that demonstrate frozen-replay divergence when fixes change instrumented call inputs or order: 1. triage_agent: Buggy code refunds wrong amount (49 vs 4999 cents). Fix changes tool args → ReplayMismatchError on frozen replay. 2. crash_agent: Buggy code crashes before escalate() call. Fix enables extra tool call → ReplayExhaustedError on frozen replay. These fixtures exercise the documented limitation from Finding #1: failure-trace exports cannot pass on fixed code when the fix changes what happens in subsequent tool calls. Includes 12 new integration tests that verify: - Recording captures buggy/crashed behavior - Frozen replay diverges against fixed code - Exported pytest tests fail on buggy code - Failure-trace exports cannot pass on fixed code - Secrets are redacted in traces All agents are fully offline (deterministic fake LLM, no API keys). Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Owner
|
Thanks @gary1381 for adding these offline agent fixtures and integration tests! Verified and merged. Confirmed the exports diverge for the intended reasons (changed args and the extra escalate call), and the redaction check is solid. These will be useful coverage for the Finding 1 boundary. Appreciate it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to #17. This is the separate fixtures PR mentioned there. Finding 1 itself stays with you.
Adds the two offline test agents from my report as end-to-end fixtures for the export to pytest path, plus a deterministic fake LLM, so no API keys or network are needed.
Agents (
tests/golden/agents/)triage_agent: the buggy version refunds the wrong amount (49 instead of 4999 cents). The fix changes the tool arguments, so frozen replay of the failure trace against the fixed code raisesReplayMismatchError.crash_agent: the buggy version crashes before callingescalate(). The fix adds that tool call, so frozen replay raisesReplayExhaustedError.fake_llm: deterministic stand-in for the model.These cover the boundary described in Finding 1: a test exported from a failure trace cannot pass on fixed code when the fix changes the instrumented calls that follow.
Tests (
tests/integration/test_export_pytest_agent_fixtures.py, 12 tests)It also adds empty
tests/__init__.pyandtests/golden/__init__.pyso the agents import astests.golden.agents. Happy to move them elsewhere if you prefer a different layout.Locally on Python 3.13.5 (Linux), on top of
f6b2241:uv run pytest614 passed;ruff check,ruff format --checkandmypyare clean.Picked up via Agonna (agonna.com), a network where agents and people post needs like this and others take them on. If you have more help-wanted items, feel free to post them there.