Repository navigation
test: adopt shared LMI lifecycle suite - #931
zhongkechen wants to merge 54 commits into
Conversation
Add opt-in cloud deployment and lifecycle coverage for Python LMI. Assert invocation deadlines, early branch cleanup, replay, and worker recovery; keep issue 741 regressions red until the SDK fix.
The root only coordinates workspace packages. Disable its implicit editable install so the LMI test directory does not trigger setuptools' multiple top-level package discovery error when starting examples. Exercise examples environment initialization in the unprivileged LMI harness job and verify that it imports the checked-out SDK.
Run the complete cloud matrix on every trusted PR update, including drafts, and every main push without labels or path filters. Preserve existing cloud credential restrictions for forks and Dependabot. Use run-owned resources without a cross-run concurrency group so newer commits cannot replace another commit's pending cloud job.
Initialize explicit hold/release objects before invoking fixtures and report missing or forbidden controls as infrastructure failures. Signal the winner only after the losing step has entered held I/O, and reject cleanup assertions when that loser exited before the winner was ready. Poll events by case with bounded parallel reads, correlate checkpoint holds to their request, and check stale effects before waiting for a retry on a potentially pinned worker. Cover the observed false positives and full fixture handshakes with regression tests.
Keep LMI on Python 3.14 with environment concurrency one and two. Exercise root finally before PENDING, bounded nested progress, concurrent shared-handler invocations, in-flight return/failure cleanup, and late operations from abandoned children using Python public APIs. Document the Java/Python mapping and preserve the newly exposed late-step scope defect as an explicit issue 741 regression.
Collect each case's own executions during teardown and leave full-run history/log collection to the final workflow step. Pace history reads and bound throttle-only retries, including reads through the public cloud runner, without retrying invocation or callback writes. Cover pagination, throttling exhaustion, permission errors, and scoped versus full-run collection with focused tests.
Checkpoint a victim's placement decision before starting fault work. If a runtime environment rotates between anchor and victim admission, replace the pair with a bounded retry. Never retry an admitted lifecycle assertion. Keep the admission decision stable during replay.
Hold a repository-wide concurrency slot through deployment, testing, and cleanup without increasing shared provider capacity. Retain up to 100 pending cloud jobs with queue: max instead of evicting older jobs. Keep the harness parallel and document the queue and repository scope.
Deploy one function per native concurrency setting and reuse it for ordinary and deadline scenarios. Run all 28 cases in one cloud job without changing provider capacity or losing regression assertions. Release controls, stop remaining executions, and observe wrapper exit before reusing each function. Extend fault observation for the shared 60-second timeout and retain a bounded emergency release.
Create the persistent stack once and update the same two functions after previous durable executions finish. Preserve failed stacks, code artifacts, and infrastructure after every test run. Isolate controls and events by run ID, expire only run data, and scope cleanup to the current run without deleting functions or buckets. Remove the infrastructure janitor and verify create/update behavior.
Keep strict cloud regressions manual while the harness runs on PRs. Pin artifact uploads to the immutable revision used on current main. Invalidate old run metadata before build and deployment preflight, preserving the build manifest and preventing prior-run cleanup.
…lmi-observability-python-20261023
This comment has been minimized.
This comment has been minimized.
| trace.emit( | ||
| "WRAPPER_RETURN", | ||
| status=result["Status"], | ||
| resources=trace.sdk_resources, | ||
| time=trace.sdk_completed_at, | ||
| ) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Updated in 268ade0. Evidence publication now records a source failure without replacing the SDK result or exception. The regression exercises both outcomes.
| contents: read | ||
|
|
||
| jobs: | ||
| lifecycle: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Maintainer agreement is still pending. The design and its limitations are documented for review.
| if on_release is not None: | ||
| on_release(released) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Updated in 268ade0. Removed the unused gate callbacks. The fixture observes the control state directly.
| elif fixture == "deadline" and scenario in { | ||
| "deadline-during-step", | ||
| "peer-recovery", | ||
| }: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Updated in 268ade0. Added parameterized coverage for deadline-during-step and peer-recovery, including cases where the native timeout and logical execution disposition differ.
This comment has been minimized.
This comment has been minimized.
| trace.emit( | ||
| "WRAPPER_RETURN", | ||
| status=result["Status"], | ||
| resources=trace.sdk_resources, | ||
| time=trace.sdk_completed_at, | ||
| ) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Updated in 268ade0. Evidence publication now records a source failure without replacing the SDK result or exception. The regression exercises both outcomes.
| elif fixture == "deadline" and scenario in { | ||
| "deadline-during-step", | ||
| "peer-recovery", | ||
| }: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Updated in 268ade0. Added parameterized coverage for deadline-during-step and peer-recovery, including cases where the native timeout and logical execution disposition differ.
| if on_release is not None: | ||
| on_release(released) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Updated in 268ade0. Removed the unused gate callbacks. The fixture observes the control state directly.
This comment has been minimized.
This comment has been minimized.
A request seal follows both the fixture wrapper and registered work. Register SDK transport calls until their exit records are published. Matches against the local shared driver confirm that a completed call can be sealed and that written late records invalidate the old seal. The first review found that writing diagnostic evidence could replace an SDK response. Later reviews reproduced the same failure during bootstrap, suspension, stored-error replay, and gate exit. One publication boundary now retains the gap and leaves the SDK result intact. Control reads and writes remain part of fixture behavior and still surface their own errors. Allocate sequence numbers at publication; retain observed times separately. Removing detached capture objects and unused scope hooks avoids unsupported paths. Let SDK initialization own pagination. The prefetch implementation used to throw a nonretryable initial-fetch error before the SDK could return its FAILED result. Delayed polls retain actual transport times. The fixture asserts delivered retry results, batches, and callback results. The staggered-overlap test exposed that a first held read could predate the other worker. Reuse the driver's existing normal gate release after both live heartbeats. Preserve BLOCKED from the first read. The driver owns peer placement; an off-target peer uses the ordinary control instead of extra placement checkpoints. A preview uses its own new directory so retained run files stay intact. Local event renumbering rebuilt records and dropped completion identity and timestamps. The regression failed before the fix because InvocationCompletedDetails disappeared when the event ID changed. Terminal StepFailed history used to expose a zero delay. The earlier test failed on an unexpected NextAttemptDelaySeconds key. The immediate retry test at b9a7a387 failed on an omitted zero for RETRY without options. Preserve explicit nonterminal delays, including the existing SUCCEED compatibility test. Updated expectations cover SDK-owned bootstrap, exact disposition of both disposal probes, late raw publication uniqueness, and publication failure through the actual trace. The old split-capture and unused scope tests exercised no production path and were replaced by ordered publication and matched-task observations. An independent publication channel or stopping a real late SDK call would change the protocol or the measured behavior. If every write after an already valid seal is lost, its consumer can still see the old seal. Tests explicitly show that blind spot. Native process termination, recovery identity before the preparation acknowledgment, and the reusable-workflow pin remain separate rollout concerns. The deployed fixture now requires a case coverage profile and owned controls. The driver judges peer and recovery placement from actual observations, with no duplicate fixture decisions. The callback result is fetched and checked even with Python optimization enabled. The registered warm worker uses the same nonthrowing publication boundary. Native overlap still uses each worker's actual normal release-read time. The gate contains no precise shared release witness. A delayed read can follow the other worker's native completion. Backdating it would create evidence the fixture did not observe. The driver and fixture will need a shared proof to close that limitation. During cleanup, resolve the requested run from a recovery journal before validating its identity. A prior-run manifest previously caused the CLI to reject the current recovery before reconciliation. Let the current runner resolve a saved case-output directory when the operator omits --output. Preserve explicit manifest and directory options and the published runner's established fallback. The local callback transport now reads actual SDK service history. Before this adaptation, the strengthened callback verifier stopped at the test transport's missing history method. The test uses fresh native request IDs, keeps service time comparable to wrapper time, and checks the completed requests against recorded wrapper entries. Rebuild and collection now stop when an unresolved deployment journal prevents proving the current verified target. Previously rebuilding could archive the journal out of the recovery path, and collection could reach AWS using the prior run's manifest. The cleanup CLI blocks conflicting journal identity before reconciliation. Completed artifacts downloaded from CI resolve case manifests relative to their local manifest; relocated absolute case paths require explicit output instead of guessing. Unresolved journals retain their original path. The runtime verifies one canonical run ID before creating its storage client. Remove pass-through observer calls and the unused fixture-owned ready-control handshake. The deployed scenarios use driver releases. Verify separate cleanup receipts for two deployment states. Recovery before archival writes a run-keyed receipt while preserving the prior active report. An active run instead writes to its saved case output. The old test expected the active destination even before archival. Direct deploy shares the unresolved-journal guard with build. Collect requires the exact function-name map that the shared deployer writes after verifying every target. Cleanup resolves symlinked manifests before locating adjacent outputs and leaves journal reconciliation identity to the shared implementation. An older pin can still clean up completed downloads beside their manifest, and manual recovery documents the run selector. The fixture verifies the exact case hash in the control prefix before opening storage. Matching aliases from a different deployment still fail identity validation. Broken diagnostic stdout cannot change a returned SDK result or replace the original SDK exception. Console diagnostics are best effort independently of S3. After a permanent pre-seal write failure, avoid another upload retry budget for each record. The first failure marks the source incomplete and leaves no valid seal. After sealing, keep attempting late writes so recovered storage can invalidate the previous seal. Resolve run and cleanup defaults from the selected manifest, but find a pending recovery journal adjacent to the supplied path if its active manifest still points elsewhere. Reject invalid deploy arguments before the pinned deployer can erase run artifacts. Preserve the requested run when rendering summaries. Use the installed runner's case-key function in fixture tests. Assert release classification on the published record and keep root-task observation owned by the root scope. Bound cumulative synchronous publication work before sealing even if several slow writes eventually succeed. Preserve late invalidation attempts after a seal. Retain capacity throughout the old pin's callback rehearsal, including earlier retry suspensions. Distinct parallel branch outputs detect incorrect reuse of the left result. Build failure summaries retain diagnostics when no manifest exists. Count retry and read-back overhead toward the evidence budget, while preserving healthy writes through a deadline-length heartbeat stream. Local terminal failure history clears retry deadlines from prior attempts. Bundle entries give regular files 0644 and executable files 0755 for Lambda extraction. No published pin or wire changes. Verify retry-to-failure and retry-to-success history reconstruction. Both regressions fail against the prior history model because the terminal step retains its first retry deadline. Audit finished bundle permissions rather than stopping before packaging. Keep warm reuse isolated from other test case IDs. Reject duplicated logical fixture bodies independently of their operation IDs. Partial missing bodies alone do not establish a duplicate. The pre-seal budget counts failed and ambiguous write attempts, while healthy first-attempt writes and post-seal invalidation remain eligible. Slow successful writes can still consume native time. Record replay duplicates as existing side-effect assertion counts. Keep collection watermarks intact so independent retirement can succeed. Count repeats per map input and scope this oracle to replay. Remove the unused gate return object and preserve checkpointed None. Persist the Python replay assertion rows with full evidence snapshots. A duplicate observed during partial collection keeps its actual body count, even when the shared builder has no sideEffects rows yet. An observed input outside the fixture map also expects zero runs. These verdicts remain independent from source coverage and retirement. Provide logical replay expectations before the shared builder saves. The paired runner applies them to complete and partial evidence and reports late behavior independently of physical retirement. The published dependency lacks this policy and retains its per-ID checks. Verify the imported conformance module comes from the exact pinned source tree, rather than accepting an older wheel in a nested venv. Require replayed map results to retain their integer value types. The shared builder gives the fixture policy validated body identities. Validate conformance checkout provenance before deployment touches persistent resources. Check SDK provenance inside the LMI Hatch environment before running its tests. Verify that the shared cleaner stops an execution recorded in a relocated case tree without rewriting the saved case manifest. Run the harness provenance assertion inside the test, including a wrong SDK import that must stop before pytest. Exercise relocated case-journal cleanup with each supported runner.
| def prepare(_): | ||
| trace.gate(preparation_gate, preparation=True) | ||
| return trace.identity["request"] |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| parameters = signature(shared_build_evidence).parameters | ||
| options: dict = {} | ||
| if "expected_replay_effects" in parameters: | ||
| options["expected_replay_effects"] = _EXPECTED_REPLAY_EFFECTS | ||
| if "replay_effect_policy" in parameters: | ||
| options["replay_effect_policy"] = replay_effects |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| delay = None | ||
| if ( | ||
| self.operation_update.operation_type == OperationType.STEP | ||
| and self.operation_update.step_options | ||
| and self.operation_update.action != OperationAction.FAIL |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
| def prepare(_): | ||
| trace.gate(preparation_gate, preparation=True) | ||
| return trace.identity["request"] |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_jxp2j535qepr5q6lj6ckg4revg
[P1] Preserve recovery ownership before preparation commits
If the original invocation is interrupted after preparation is released but before its SUCCEED checkpoint persists, the default at-least-once semantics rerun prepare() on the next invocation. This returns the retry's request ID, admitting that retry into the victim fault and admission probes. The supplied regression interrupts only after preparation commits. Read and verify the immutable driver-owned request selection before admitting preparation, checkpoint that selected identity, and add a regression covering interruption before the preparation checkpoint.
To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.
| parameters = signature(shared_build_evidence).parameters | ||
| options: dict = {} | ||
| if "expected_replay_effects" in parameters: | ||
| options["expected_replay_effects"] = _EXPECTED_REPLAY_EFFECTS | ||
| if "replay_effect_policy" in parameters: | ||
| options["replay_effect_policy"] = replay_effects |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_tad5hccecyxnngwp33vgugt66z
[P2] Return to one supported evidence contract
Rule 2, Simplify before you add, remains unmet: this negotiates multiple reader APIs even though verify_checkout() permits exactly one revision. Cleanup and summary contain similar compatibility paths without explaining why one fixed interface cannot work. The committed pin also bypasses the new replay policy, and consumer tests conditionally skip unsupported checks. Return to the design phase (rule 8, step 6) and agree on one validated producer/reader contract; binding directly to that contract is the simpler direction.
To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.
| delay = None | ||
| if ( | ||
| self.operation_update.operation_type == OperationType.STEP | ||
| and self.operation_update.step_options | ||
| and self.operation_update.action != OperationAction.FAIL |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_a5yghnygi3ci3lqbcdpztaehck
[P2] Include the testing-model fixes in the root-cause account
Rule 5, A bug fix starts with a failing test, requires the description to name each bug's root cause. The description explains fixture and evidence-protocol problems but omits these shipped testing-library fixes: terminal events retaining retry metadata and event-ID reassignment dropping invocation completion details. These independently affect public history results. Return to the design phase (rule 8, step 6) and include their causes and reproducing tests in the agreed scope.
To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.
Codex AI reviewThree findings: replay ownership ( Reviewed commit |
This adapts the Python SDK to the seven lifecycle requirements and reusable harness in conformance #186, pinned to
41878367192be996cb3414939e15270febedb210.Design note for the shared producer refactor
Agreed before this refactor: centralize the reusable fixture producer in the conformance stack, remove the unreleased no-coverage fixture mode, and move the deadline/recovery victim execution-status policy into the shared normalizer. Independent Codex and Claude design analyses completed; the maintainer selected the two behavior changes where they differed.
The Lambda ZIP will contain the core SDK, its resolved dependencies, fixture/hook modules and only the allowlisted shared runtime leaf plus minimal namespace/legal files. It will exclude the conformance runner, catalogs, YAML/report stack and optional OTel dependencies. The independent pre-install exact-Git-pin guard remains in the caller.
Merge order: #187 → #188 → #189 → #190 → #931. Update every caller pin to the same remotely available final shared revision. Implementation and its validation are in progress; the validation below describes the existing pinned revision until it is replaced with the final results.
Change
.github/workflows/lmi-e2e-tests.ymlcalls the sharedlmi-lifecycle.ymlat that exact SHA. The shared workflow owns harness/cloud jobs, AWS credentials and account checks, deployment/test budgets, collection, cleanup, artifacts and summary..github/scripts/lmi-ci.shsupplies Python setup/build/test commands. GitHub Actions' fixed cloud-job concurrency group remains the sole cross-run guard, covering deployment through cleanup; no S3 shared lock is used.Reusable AWS transport, deployment, scenario orchestration, evidence normalization, reporting, cleanup and their regression tests moved to #186. Python retains its real public SDK fixtures, lifecycle observation hooks, core wheel/ZIP build, bootstrap and thin runtime bindings. The Lambda bundle contains only the checked-out core SDK, its dependencies and fixture/observation modules. Production SDK behavior is unchanged.
All seven cases are enabled in their default order:
lmi-lifecycle-1native-overlaplmi-lifecycle-2normal-returnlmi-lifecycle-3deadline-during-steplmi-lifecycle-4peer-recoverylmi-lifecycle-5suspension-retrylmi-lifecycle-6warm-reuselmi-lifecycle-7normal-failureThe adapter reads the shared matrix directly. Identity is run/case based, without
cohort_idorcohortId. Probes after both wrapper return and raise preserve actual SDK admission behavior. Case 4 checkpoints the victim's original request ID, admits the peer halfway through the victim timeout, verifies the remaining recovery window and excludes expired originals or service retries from healthy-peer evidence. Retries do not reinject the victim fault. Completion timestamps are captured once in milliseconds and reused without conversion drift.Decisions not covered by the design note
One line each, for maintainer acceptance (rule 7):
cleanupMsbudget.late_activitywhile the warm invocation is still running, which the validator requires.admissionMs(default 5 s). Each probe record is a separate S3 publication and the runtime freezes the environment when the handler returns; the earlier 0.5 s join could truncate the ledger.observation_error, never as an SDK outcome. The backend's checkpoint or state result, or its error, is returned unchanged.skip_time=False). The default virtual clock skips the fixture's durable wait and dates service events ahead of the wall-clock retirement they are compared against.Validation
At
4187836, with this branch's head:lmi_testsunit and local e2e: 70 passed, including real public-SDK replay, normal return/failure, warm reuse and recovery instrumentation.ObservedClient, and each condition of the deadline cooperative exit was checked by removing it from the fixture.hatch run lmi:lintand thelint-commitsscript over the PR range: passed locally. Pinned checkout verified withverify_checkout.4187836: 2,005 tests passed, ruff and mypy clean.4187836: scoped mypy, Bash syntax and stage tests, and the Python 3.14 arm64 Lambda bundle build and inspection.Unverified (rule 6): whole-suite conformance against real LMI. The latest completed cloud run, at the earlier pin
793ecdc, run 43956550294, passed case 1; cases 2 and 4 failed on incomplete evidence, and case 2's unretired owner quarantined the c1 function, so cases 3, 5, 6 and 7 did not run cleanly. That run predates the evidence-publication isolation fix. Existing late-admission behavior remains visible as failure: an asynchronous checkpoint can be accepted after normal completion and a late step can remain pending.See run instructions and suite ownership. Merge #186 before #931; retain the exact shared revision pin.
Agreed redesign after the shared-producer review
Before further code, independent Codex and Claude designs revisited the new findings under CONTRIBUTING rules 1–9. The maintainer approved placement being owned by the shared original-request environment checks, and retention of the necessary global Hatch skip-install flag with its workspace effects documented.
The fixture placement-admission checkpoint and unused raw placement phases will be removed. This changes an unreleased fixture operation sequence, so a newly built ZIP/configuration identity, new run/execution namespace and independently completed retirement isolate older checkpoint histories. Misplaced healthy peer/recovery work remains case-owned and retired; the victim fault path and public SDK behavior stay unchanged.
Other corrections narrow interruption handling, retain the actual execution ErrorObject in local evidence, report absent durable context as an observation error rather than fake admission probes, remove inactive observer APIs and stale cleanup/invocation branches, and scope PR triggers to relevant main-branch inputs. Real public hook, operation/task, map/parallel/replay coverage remains.
Actual CI-pinned Hatch1.16.5 testing showed removing global skip-install causes root installation to fail with two discovered top-level packages, packages and lmi_tests. The flag will remain; its effect on the default and eight inherited environments is an explicit accepted workspace decision. Detached LMI behavior is independently verified. IO_CONFIG, timing, toolchain and trusted checkout override choices will be documented with tests and unverified native assumptions.
Implementation and final rebased/shared-pin validation are in progress. The older validation section is historical until replaced with exact final-head results; a provisional shared pin will not be published before that commit is remotely available.
Approved evidence protocol design
The source collector currently compares controller, fixture and service timestamps as if they came from one clock. A contiguous ledger can also lose its last event, and an interruption before the preparation checkpoint commits can let a retry inject a fault intended for the original invocation. These can change a conformance verdict or incorrectly return a target to the next case.
The proposed raw protocol revision keeps the existing runner entry points. Each new run pins its SDK artifact and collector revision. Historical evidence stays with its original reader. The shared driver records clock domains, capture order, causal receipts, outstanding work and a positive end-of-stream record. Same-domain timing and bounded cross-domain intervals can establish the manifest budgets, including equality. Ambiguous timing is incomplete. A recorded late transport handoff or fresh post-return body is a behavioral failure. A backend rejection preserves the fact that a handoff occurred. At retirement the driver needs evidence covering every owner through the cutoff. A successful operation, native timeout or generic throttle alone cannot establish that coverage. A target with independently confirmed physical retirement can be reused even when the evidence cannot establish a behavioral budget.
Python will exercise the public SDK, bind early observations after the SDK resolves paginated input, and compare delivered results and caught failures with service history by operation ID. The regression cases include a stored failure returned as a value, an extra nested body with unchanged output, loss of a terminal publication, and a retry on either side of the preparation checkpoint. In a local counterexample the current fixture raised
StopIterationwith the root on page two and made zero page requests. Actual cloud acceptance of the proposed revision is unverified.Original request ownership. The maintainer selected driver-owned admission. Before releasing preparation, the shared driver will persist an immutable case-owned record that identifies the launch ARN and selected native request, then verify the stored record. Only the selected request may enter victim work or start its probes. If a retry resumes before the preparation checkpoint commits, it reads the same selection. Ambiguous identity yields incomplete evidence. A producer that has finished can publish a final sequence after accounting for its work. A peer that is still held can acknowledge the observation cutoff. If a hard kill leaves user work unaccounted for, the target stays fenced pending independently recorded physical retirement.
This is the agreed design under contribution rule 8. The shared implementation belongs in conformance #189 and the Python producer belongs here. The mandatory post-closure probes and the existing validator verdict still apply.