You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(minimize): refuse a signature produced before any recorded call - #23
Implements the rule you accepted in #22, with your two notes applied. Closes#22.
refuse when consumed == 0 and calls_before_recorded_error >= 1 and no divergence
exit 2, message names both readings, write no bundle
What changed
minimize/signature.py
ReplayNeverReachedRecordedCall, a FailureDetectionError subclass, so existing callers that already catch the parent keep working.
The attempt now reads session.divergence and len(session.calls) - session.pendingafter the try/except chain instead of only on the success path (your note 2). Same for the policy-refusal check that used to read a divergent-but-assigned value.
_calls_before_recorded_failure(trace) counts llm_request/tool_call events before the trace's own error event, so the message can name what it expected: "the trace records its failure after 2 dependency call(s)".
_describe_attempt names what actually happened (ModuleNotFoundError: No module named ... / ValueError: config file missing), so the refusal carries its own evidence.
The guard sits after the attempt and before signature dispatch, so it covers the exception path and the output-mismatch path alike.
minimize/engine.py re-raises the refusal instead of folding it into MinimizationResult(success=False); cli/minimize.py maps it to exit 2. Every other detection failure still exits 1, and the ReplayPolicyError refusals from #19 are untouched (divergence is set there, so the guard cannot fire).
docs/cli.md lists the new exit-2 case and its message.
Acceptance arms, run on this branch
Same fixture as the #22 table: a trace recorded with record() (llm → lookup_customer → ValueError, so the recorded failure follows 2 dependency calls).
✗ Cannot minimize: the replay failed before reaching any recorded call
(ModuleNotFoundError: No module named 'definitely_not_installed_module'), and
the trace records its failure after 2 dependency call(s): this run never
reached the recorded failure. Either the fixture could not be built, or the
change moved the failure earlier than the recording. No bundle written.
Arm E (a genuine regression that fails earlier) is refused too, as agreed — it and the setup error are the same observation from inside the replay.
Tests
tests/unit/test_minimize_unreached_call.py, 8 tests: both setup arms refused, the refusal staying inside FailureDetectionError, the CLI writing no bundle and exiting 2, a same-calls-different-type failure still minimizing, the CLI still minimizing it, "returns without failing" keeping exit 1, and a trace whose recorded error precedes every call not being refused.
Full suite: 568 passed, 4 failed — the four are the LangGraph/model-decision integration tests, which fail on main too with ModuleNotFoundError: No module named 'langchain_core' (the optional dependency is not installed in this venv). tests/unit alone: 503 passed before the change, 511 after.
mypy --config-file pyproject.toml over the changed modules reports only import-not-found for typer (not installed in the interpreter running mypy), no errors in the changed code. I could not run ruff in this environment — pip install ruff timed out at 10 minutes, and the cached binary is killed by macOS — so the lint step is CI's to confirm, as it was on the last two PRs.
One correction to the body's verification paragraph, since it was wrong as first written and the CI run is public anyway.
The first push failed MyPy on all three versions. Two errors were real and in the new test file (Returning Any from function declared to return dict[str, str], and a helper with no return annotation); the third could only have come from a full run — the arm that tests a missing dependency did a literal import definitely_not_installed_module, which strict mypy reports as import-not-found. That arm now uses importlib.import_module(...), which is also the more honest spelling of "this module is meant to be absent". All three are fixed in 681795b, and CI is green on 3.11 / 3.12 / 3.13.
My local mypy run had missed them because I pointed it at a narrower scope without --python-executable .venv/bin/python, so it never resolved the venv's packages. Re-run the way CI does it, the changed modules are clean. ruff still has not run here — pip install ruff timed out — so lint remains CI's to confirm.
Thanks @how2how2how2-arch for opening this PR and fixing the issue. Merged!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the rule you accepted in #22, with your two notes applied. Closes #22.
What changed
minimize/signature.pyReplayNeverReachedRecordedCall, aFailureDetectionErrorsubclass, so existing callers that already catch the parent keep working.session.divergenceandlen(session.calls) - session.pendingafter the try/except chain instead of only on the success path (your note 2). Same for the policy-refusal check that used to read a divergent-but-assigned value._calls_before_recorded_failure(trace)countsllm_request/tool_callevents before the trace's ownerrorevent, so the message can name what it expected: "the trace records its failure after 2 dependency call(s)"._describe_attemptnames what actually happened (ModuleNotFoundError: No module named .../ValueError: config file missing), so the refusal carries its own evidence.The guard sits after the attempt and before signature dispatch, so it covers the exception path and the output-mismatch path alike.
minimize/engine.pyre-raises the refusal instead of folding it intoMinimizationResult(success=False);cli/minimize.pymaps it to exit 2. Every other detection failure still exits 1, and theReplayPolicyErrorrefusals from #19 are untouched (divergence is set there, so the guard cannot fire).docs/cli.mdlists the new exit-2 case and its message.Acceptance arms, run on this branch
Same fixture as the #22 table: a trace recorded with
record()(llm →lookup_customer→ValueError, so the recorded failure follows 2 dependency calls).Exception: ModuleNotFoundError, exit 0, 3-event bundleException: ValueError, exit 0Exception: KeyError, exit 0Exception: ValueError, exit 0Exception: TypeError, exit 0Refusal message:
Arm E (a genuine regression that fails earlier) is refused too, as agreed — it and the setup error are the same observation from inside the replay.
Tests
tests/unit/test_minimize_unreached_call.py, 8 tests: both setup arms refused, the refusal staying insideFailureDetectionError, the CLI writing no bundle and exiting 2, a same-calls-different-type failure still minimizing, the CLI still minimizing it, "returns without failing" keeping exit 1, and a trace whose recorded error precedes every call not being refused.Full suite: 568 passed, 4 failed — the four are the LangGraph/model-decision integration tests, which fail on
maintoo withModuleNotFoundError: No module named 'langchain_core'(the optional dependency is not installed in this venv).tests/unitalone: 503 passed before the change, 511 after.mypy --config-file pyproject.tomlover the changed modules reports onlyimport-not-foundfortyper(not installed in the interpreter running mypy), no errors in the changed code. I could not runruffin this environment —pip install rufftimed out at 10 minutes, and the cached binary is killed by macOS — so the lint step is CI's to confirm, as it was on the last two PRs.