Skip to content

test(lmi): reproduce gaps in held-fault recovery evidence - #199

Draft
zhongkechen wants to merge 1 commit into
codex/lmi-peer-dispatch-reprofrom
codex/lmi-held-fault-recovery-repro
Draft

zhongkechen wants to merge 1 commit into
codex/lmi-peer-dispatch-reprofrom
codex/lmi-held-fault-recovery-repro

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

The lmi-lifecycle-4 validator accepts two runs that do not establish recovery while the victim’s external fault is held:

  • The victim exits at 35 ms and the worker reports recovery at 45 ms. Releasing the fault at 50 ms passes, even though the selected recovery request does not succeed on the original target until 60 ms. Releasing at 70 ms is a valid control; the healthy peer stays active until 100 ms.
  • The victim waits on a different gate from the one in the fault inventory. The recorded fault stays held, but it cannot establish what happened to the victim’s gate.

Both cases pass observation and final validation today. The controls also retain the Python fixture’s own deadline observation: it cannot substitute for the platform timeout or the victim task’s exit. No SDK cancellation signal is assumed.

The fix also needs to reject an empty fault inventory. On #193, clearing both faultsCreated and retirement.faults lets lifecycle 4 pass observation and final validation. The draft does not yet include this regression. Add coverage in both phases and require a fault record tied to the victim’s gate that stays held until the selected recovery request succeeds. A nonempty inventory alone cannot establish that link. Review comment

Recovery policy

For profiles that run lifecycle 4, require the manifest to declare the expected recovery_policy (reuse-worker or replace-worker) and have the CLI reject evidence that reports a different policy. In the lifecycle-4 evidence schema branch, require profile.recoveryPolicy to match the semantic validator. Add regression tests for a missing manifest policy, a policy mismatch, and missing evidence policy. Manifest review comment · Schema review comment

This draft follows up on point 3 in #193. It changes only the evidence tests; the fix will come later. Stack: #193 → #195 → #189 → #190 → this draft.

To reproduce both gaps, run from the repository root:

hatch run test:all packages/aws-durable-execution-conformance-tests/tests/test_lmi_evidence.py -k held_fault_recovery --runxfail

That run produces four failures and seven passing controls. With the four regressions marked as strict expected failures, the full workspace suite had 2,108 passes. Type, format, lint, YAML, and distribution checks passed. These are offline validator tests; no live LMI invocation was run.

@zhongkechen
zhongkechen changed the base branch from feat/lmi-4-workflow to codex/lmi-peer-dispatch-repro November 18, 2026 18:58
@zhongkechen
zhongkechen added this pull request to stack #196 November 18, 2026 18:58
@zhongkechen
zhongkechen force-pushed the codex/lmi-held-fault-recovery-repro branch from 584254d to 43abab3 Compare November 20, 2026 02:06
@zhongkechen

Copy link
Copy Markdown
Contributor Author

/ai review

Comment on lines +290 to +292
if self.scenario == "peer-recovery":
# Delayed history publication does not extend the ledger watermark.
self.cloud.recovery_completion(self.items["recovery"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review · Finding arf_v1_55rdb2wjuqyryg4geaulodlnc6

[P2] Wait for native recovery completion before releasing the peer

This wait happens after _deadline() has already released and finished the healthy peer. Cloud.finish(recovery) waits for logical execution success and wrapper return, but does not require the original request's native completion. If that completion occurs after the released peer finishes, the new validator rejects the run even though the harness caused the capacity gap. Move the completion wait into _deadline(), before its final peer-liveness check and peer release. Add a regression where native recovery completion lags logical success and wrapper return.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

Comment on lines +2052 to +2056
remaining = iter(complete)
return all(
any(
all(
key not in fragment or same_json(fragment[key], receipt.get(key))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review · Finding arf_v1_e22ajchen6d2cppwuohokuomqw

[P2] Match completed receipts independently of array order

The shared iterator makes receipt matching order-dependent. For partial receipts [A, B] from different requests and completed receipts [B, A] with matching identities, searching for A consumes B, which is then incorrectly reported missing. Cross-request array order is not an evidence identity, so this rejects valid cleanup and quarantines the profile solely because a collector reordered records. Use one-to-one identity matching independent of array position, preserving multiplicity, and add a regression with reordered completed receipts.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

Two correctness findings affect recovery sequencing and cleanup receipt matching. Review was static; tests and live LMI validation were not run.

Reviewed commit 43abab34a0ff1b259d581ca50b011c73fcf47dd3. Workflow run

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant