Skip to content

fix(minimize): refuse a signature produced before any recorded call - #23

Merged
utsab345 merged 6 commits into
utsab345:mainfrom
how2how2how2-arch:minimize-unreached-call
Oct 19, 2026
Merged

utsab345 merged 6 commits into
utsab345:mainfrom
how2how2how2-arch:minimize-unreached-call

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Implements the rule you accepted in #22, with your two notes applied. Closes #22.

refuse when consumed == 0 and calls_before_recorded_error >= 1 and no divergence
exit 2, message names both readings, write no bundle

What changed

minimize/signature.py

  • ReplayNeverReachedRecordedCall, a FailureDetectionError subclass, so existing callers that already catch the parent keep working.
  • The attempt now reads session.divergence and len(session.calls) - session.pending after the try/except chain instead of only on the success path (your note 2). Same for the policy-refusal check that used to read a divergent-but-assigned value.
  • _calls_before_recorded_failure(trace) counts llm_request/tool_call events before the trace's own error event, so the message can name what it expected: "the trace records its failure after 2 dependency call(s)".
  • _describe_attempt names what actually happened (ModuleNotFoundError: No module named ... / ValueError: config file missing), so the refusal carries its own evidence.

The guard sits after the attempt and before signature dispatch, so it covers the exception path and the output-mismatch path alike.

minimize/engine.py re-raises the refusal instead of folding it into MinimizationResult(success=False); cli/minimize.py maps it to exit 2. Every other detection failure still exits 1, and the ReplayPolicyError refusals from #19 are untouched (divergence is set there, so the guard cannot fire).

docs/cli.md lists the new exit-2 case and its message.

Acceptance arms, run on this branch

Same fixture as the #22 table: a trace recorded with record() (llm → lookup_customer → ValueError, so the recorded failure follows 2 dependency calls).

entrypoint before after
import error Exception: ModuleNotFoundError, exit 0, 3-event bundle refused, exit 2, no bundle
setup error raising the recorded type Exception: ValueError, exit 0 refused, exit 2, no bundle
early failure, other type Exception: KeyError, exit 0 refused, exit 2, no bundle
recorded failure reproduced (2/2 consumed) Exception: ValueError, exit 0 unchanged, exit 0, bundle written
all calls consumed, different type (2/2) Exception: TypeError, exit 0 unchanged, exit 0, bundle written
returns without failing exit 1, no bundle unchanged, exit 1
trace whose error precedes every call — not refused (guard degenerates to false)

Refusal message:

✗ Cannot minimize: the replay failed before reaching any recorded call
(ModuleNotFoundError: No module named 'definitely_not_installed_module'), and
the trace records its failure after 2 dependency call(s): this run never
reached the recorded failure. Either the fixture could not be built, or the
change moved the failure earlier than the recording. No bundle written.

Arm E (a genuine regression that fails earlier) is refused too, as agreed — it and the setup error are the same observation from inside the replay.

Tests

tests/unit/test_minimize_unreached_call.py, 8 tests: both setup arms refused, the refusal staying inside FailureDetectionError, the CLI writing no bundle and exiting 2, a same-calls-different-type failure still minimizing, the CLI still minimizing it, "returns without failing" keeping exit 1, and a trace whose recorded error precedes every call not being refused.

Full suite: 568 passed, 4 failed — the four are the LangGraph/model-decision integration tests, which fail on main too with ModuleNotFoundError: No module named 'langchain_core' (the optional dependency is not installed in this venv). tests/unit alone: 503 passed before the change, 511 after.

mypy --config-file pyproject.toml over the changed modules reports only import-not-found for typer (not installed in the interpreter running mypy), no errors in the changed code. I could not run ruff in this environment — pip install ruff timed out at 10 minutes, and the cached binary is killed by macOS — so the lint step is CI's to confirm, as it was on the last two PRs.

@how2how2how2-arch

Copy link
Copy Markdown
Contributor Author

One correction to the body's verification paragraph, since it was wrong as first written and the CI run is public anyway.

The first push failed MyPy on all three versions. Two errors were real and in the new test file (Returning Any from function declared to return dict[str, str], and a helper with no return annotation); the third could only have come from a full run — the arm that tests a missing dependency did a literal import definitely_not_installed_module, which strict mypy reports as import-not-found. That arm now uses importlib.import_module(...), which is also the more honest spelling of "this module is meant to be absent". All three are fixed in 681795b, and CI is green on 3.11 / 3.12 / 3.13.

My local mypy run had missed them because I pointed it at a narrower scope without --python-executable .venv/bin/python, so it never resolved the venv's packages. Re-run the way CI does it, the changed modules are clean. ruff still has not run here — pip install ruff timed out — so lint remains CI's to confirm.

@utsab345
utsab345 merged commit f6b2241 into utsab345:main Oct 19, 2026
6 checks passed
@utsab345

Copy link
Copy Markdown
Owner

Thanks @how2how2how2-arch for opening this PR and fixing the issue. Merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

minimize: refuse signatures produced before any recorded call is consumed

2 participants