Repository navigation
test: adopt shared LMI lifecycle suite #931
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zhongkechen
wants to merge
54
commits into
main
Choose a base branch
from
feat/python-lmi-shared-suite
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+6,034
−39
Open
Changes from all commits
Commits
Show all changes
54 commits
Select commit
Hold shift + click to select a range
384529a
test: add LMI cloud lifecycle regressions
zhongkechen f93a3c8
fix: skip installing the workspace root
zhongkechen d8005d2
ci: run LMI tests on PRs and main
zhongkechen 70e24c6
fix: repair LMI control barriers and evidence
zhongkechen 9150c87
test: add Java lifecycle guards on Python 3.14
zhongkechen 3ebcf7d
test: pace LMI execution history reads
zhongkechen ef13146
test: bound LMI peer placement before admission
zhongkechen f385a1d
ci: serialize LMI cloud jobs across runs
zhongkechen 31d9a30
test: share LMI functions in one deployment
zhongkechen 7bd33b1
test: retain LMI functions across runs
zhongkechen 9e0564f
Merge main into Python LMI cloud tests
zhongkechen ab723ef
fix: isolate LMI runs and gate cloud regressions
zhongkechen 1fd8900
chore: merge latest upstream main
045ed56
Merge branch 'main' into test/python-lmi-e2e
zhongkechen 558892b
Merge commit 'refs/codex/lmi-observability-20261023/main' into codex/…
zhongkechen d2c5262
test: add LMI observability coverage
zhongkechen 00949ab
Merge branch 'main' into test/python-lmi-e2e
zhongkechen de4588d
ci: refresh credentials for LMI test phases
zhongkechen b691585
test: preserve the observed SDK return boundary
zhongkechen 3aeb6a5
Merge branch 'main' into test/python-lmi-e2e
zhongkechen d31f074
Merge branch 'main' into test/python-lmi-e2e
zhongkechen 2179e01
Merge branch 'main' into test/python-lmi-e2e
zhongkechen aaaf2b6
docs(lmi): describe shared conformance adapter design
zhongkechen b2412d5
Merge branch 'main' into feat/python-lmi-shared-suite
zhongkechen 8d1308c
test: adapt Python LMI fixtures to shared conformance
zhongkechen c6c9db0
ci: run LMI PR tests on CodeBuild
zhongkechen d835bcc
fix(lmi): consume corrected conformance evidence
zhongkechen 33fc193
test(lmi): own Python-specific regression contracts
zhongkechen fda7daa
test: adopt shared LMI lifecycle suite
zhongkechen ecb0cb3
fix: reuse LMI completion timestamp
zhongkechen 2c8162b
fix: handle unacquired LMI run leases
zhongkechen 0996b2e
refactor: use Actions for LMI run concurrency
zhongkechen ac35090
fix: preserve the original recovery peer
zhongkechen e148ac8
test: temporarily skip lifecycle case 2
zhongkechen c0eb43a
fix: record recovery cohort admission
zhongkechen 45d6a43
fix: scope recovery faults to the original request
zhongkechen a1d43d4
test: prioritize case 7 during diagnostics
zhongkechen 014bb17
fix: observe admission after exceptional exit
zhongkechen 58bf505
test: restore lifecycle case 2
zhongkechen e428872
refactor: remove redundant cohort ID
zhongkechen 044fbd8
refactor: consume shared LMI workflow and driver
zhongkechen 1b0093e
chore: update shared LMI fixes pin
ceae5d1
fix(lmi): wait for the declared admission budget
zhongkechen 7fe54bd
Merge branch 'feat/python-lmi-shared-suite' into pr-931
zhongkechen db4313e
test(lmi): repin the shared conformance revision
zhongkechen 4eeb4ed
test(lmi): use the real clock for replay evidence
zhongkechen 8ac1528
test(lmi): repin the shared conformance revision
zhongkechen 717be40
fix(lmi): isolate evidence publication from SDK calls
zhongkechen 2d458e9
test(lmi): cover the deadline cooperative exit
zhongkechen 5bbf980
test(lmi): repin the shared conformance revision
zhongkechen 7bdd930
Merge branch 'main' into feat/python-lmi-shared-suite
zhongkechen d08f256
Merge branch 'main' into feat/python-lmi-shared-suite
zhongkechen 268ade0
test(lmi): seal native request evidence
zhongkechen 5fc72e8
Merge commit 'e7c09658bcf95ecab27d0d0c565d11b21dc5883e' into lmi931-l…
zhongkechen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/usr/bin/env bash | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # Python commands for the shared LMI reusable workflow's stage contract. | ||
| set -euo pipefail | ||
|
|
||
| case "${1:?Expected a workflow stage}" in | ||
| setup) | ||
| uv pip install hatch==1.16.5 | ||
| hatch run lmi:shared-setup | ||
| ;; | ||
| harness) | ||
| hatch run lmi:lint | ||
| hatch run lmi:python - <<'PYTHON' | ||
| from pathlib import Path | ||
| import aws_durable_execution_sdk_python as sdk | ||
|
|
||
| source = Path('packages/aws-durable-execution-sdk-python/src').resolve() | ||
| assert Path(sdk.__file__).resolve().is_relative_to(source), sdk.__file__ | ||
| PYTHON | ||
| hatch run lmi:shared-tests --junitxml=lmi_tests/artifacts/unit.xml | ||
| ;; | ||
| build) hatch run lmi:build ;; | ||
| deploy) | ||
| hatch run lmi:python -m lmi_tests.deploy deploy \ | ||
| --run-id "$LMI_SHARED_RUN_ID" \ | ||
| --runtime "$LMI_RUNTIME" --stack-name "$LMI_STACK_NAME" | ||
| ;; | ||
| run) hatch run lmi:shared ;; | ||
| collect) hatch run lmi:python -m lmi_tests.deploy collect ;; | ||
| cleanup) hatch run lmi:shared-cleanup ;; | ||
| summary) hatch run lmi:python -m lmi_tests.summary ;; | ||
| *) printf 'Unknown LMI workflow stage: %s\n' "$1" >&2; exit 2 ;; | ||
| esac |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| name: LMI e2e tests | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| types: [opened, synchronize, reopened] | ||
| branches: [main] | ||
| paths: | ||
| - '.github/workflows/lmi-e2e-tests.yml' | ||
| - '.github/scripts/lmi-ci.sh' | ||
| - 'lmi_tests/**' | ||
| - '!lmi_tests/**/*.md' | ||
| - 'packages/aws-durable-execution-sdk-python/src/**' | ||
| - 'packages/aws-durable-execution-sdk-python/pyproject.toml' | ||
| - 'packages/aws-durable-execution-sdk-python-testing/src/**' | ||
| - 'packages/aws-durable-execution-sdk-python-testing/pyproject.toml' | ||
| - 'pyproject.toml' | ||
| workflow_dispatch: | ||
| inputs: | ||
| region: | ||
| description: Region containing the test LMI capacity provider | ||
| default: us-west-2 | ||
| type: string | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| lifecycle: | ||
|
zhongkechen marked this conversation as resolved.
Resolved
zhongkechen marked this conversation as resolved.
Resolved
zhongkechen marked this conversation as resolved.
Resolved
zhongkechen marked this conversation as resolved.
Resolved
zhongkechen marked this conversation as resolved.
Resolved
This comment was marked as outdated.
Sorry, something went wrong.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maintainer agreement is still pending. The design and its limitations are documented for review. |
||
| permissions: | ||
|
zhongkechen marked this conversation as resolved.
Resolved
|
||
| contents: read | ||
| id-token: write | ||
| uses: aws/aws-durable-execution-conformance-tests/.github/workflows/lmi-lifecycle.yml@41878367192be996cb3414939e15270febedb210 | ||
| with: | ||
| conformance-ref: 41878367192be996cb3414939e15270febedb210 | ||
| sdk-script: .github/scripts/lmi-ci.sh | ||
| runtime-label: Python 3.14 | ||
| lambda-runtime: python3.14 | ||
| stack-name: python-lmi-e2e | ||
| region: ${{ inputs.region || 'us-west-2' }} | ||
| secrets: | ||
| TEST_ROLE_ARN: ${{ secrets.TEST_ROLE_ARN }} | ||
| TEST_ACCOUNT_ID: ${{ secrets.TEST_ACCOUNT_ID }} | ||
| TEST_LAMBDA_EXECUTION_ROLE_ARN: ${{ secrets.TEST_LAMBDA_EXECUTION_ROLE_ARN }} | ||
| CAPACITY_PROVIDER_ARN: ${{ secrets.CAPACITY_PROVIDER_ARN }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -38,3 +38,5 @@ dist/ | |
| **/template.generated.json | ||
|
|
||
| .env | ||
|
|
||
| /.conformance/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| artifacts/ | ||
| build/ | ||
| __pycache__/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,170 @@ | ||
| # Python LMI lifecycle conformance | ||
|
|
||
| This PR adapts the Python SDK to the seven shared lifecycle requirements in | ||
| [conformance #186](https://github.com/aws/aws-durable-execution-conformance-tests/pull/186). | ||
| The exact dependency is pinned in `shared_contract.py` and the workflow. Each | ||
| shared requirement declares one scalar `scenario`; the Python adapter reads the | ||
| shared matrix directly rather than maintaining a second requirement catalog. | ||
|
|
||
| | Requirement | Scenario | Target | Invocation roles | | ||
| | --- | --- | --- | --- | | ||
| | `lmi-lifecycle-1` | `native-overlap` | c2 | victim, peer | | ||
| | `lmi-lifecycle-2` | `normal-return` | c1 | victim | | ||
| | `lmi-lifecycle-3` | `deadline-during-step` | c1 | victim | | ||
| | `lmi-lifecycle-4` | `peer-recovery` | c2 | victim, peer, recovery | | ||
| | `lmi-lifecycle-5` | `suspension-retry` | c1 | victim, resume | | ||
| | `lmi-lifecycle-6` | `warm-reuse` | c1 | victim, warm | | ||
| | `lmi-lifecycle-7` | `normal-failure` | c1 | victim | | ||
|
|
||
| The shared package owns assertions, AWS transport, persistent deployment, scenario | ||
| choreography, source normalization, strict reports, scheduling and retirement. | ||
| This repository owns the real Python SDK handlers, observation hooks, ZIP build | ||
| and thin runtime bindings. The workflow definition also lives in the shared | ||
| repository; `.github/workflows/lmi-e2e-tests.yml` calls its pinned | ||
| `lmi-lifecycle.yml`, using `.github/scripts/lmi-ci.sh` for Python stage commands. | ||
| OTel/exporter tests and the previous Python-specific matrix are deferred. | ||
| There is no local conformance profile or validator that replaces a shared rule. | ||
|
|
||
| ## Local validation | ||
|
|
||
| Install Hatch and Python 3.14. Clone the shared repository into `.conformance` | ||
| and check out the exact `CONFORMANCE_REVISION` from `shared_contract.py`: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/aws/aws-durable-execution-conformance-tests.git .conformance | ||
| git -C .conformance checkout --detach 41878367192be996cb3414939e15270febedb210 | ||
| hatch run lmi:shared-setup | ||
| hatch run lmi:lint | ||
| hatch run lmi:shared-tests | ||
| hatch run lmi:build | ||
| ``` | ||
|
|
||
| `shared-setup` verifies the commit and rejects edited shared package sources. | ||
| It installs only the shared core runner. The LMI environment contains the checked | ||
| out core SDK and testing library; the Lambda bundle contains the core SDK and | ||
| its dependencies plus fixture/observation modules. It does not include the OTel | ||
| SDK, an OTel plugin, local catalogs, validators or the shared runner itself. | ||
|
|
||
| Python unit tests cover pinned workflow delegation, the Python artifact/manifest | ||
| binding, fixture observations and bootstrap. Shared driver regression tests now | ||
| live in the conformance repository and cover case ownership, cancellation, source | ||
| normalization, driver ordering and cleanup. | ||
|
|
||
| A local plan preview defaults to `lmi_tests/artifacts/preview` and requires a | ||
| new output directory. Run and cleanup use `lmi_tests/artifacts/shared` by default. | ||
|
|
||
| Local e2e tests use the actual public | ||
| SDK and its in-process test service, replacing only AWS transport and evidence | ||
| storage. They verify checkpoint/replay and preserve genuine admission failures. | ||
| These controls do not establish real LMI placement or deployed acceptance. | ||
|
|
||
| ## Cloud execution | ||
|
|
||
| Use a dedicated test account with the configured LMI capacity provider and Lambda | ||
| execution role. The workflow validates the account before deployment. The | ||
| persistent functions use Python 3.14, arm64, 2048 MiB and native concurrency 1 and | ||
| 2, with one environment each. SDK artifact and configuration digests are retained | ||
| in every manifest and checked before invoking either target. | ||
|
|
||
| Run cloud tests through the workflow's `workflow_dispatch` action or a trusted | ||
| same-repository PR. Its fixed `lmi-e2e-shared-capacity-provider` concurrency group | ||
| serializes cloud jobs across PRs and refs from deployment through final cleanup, | ||
| with `cancel-in-progress: false`. GitHub Actions is the only cross-run concurrency | ||
| control. Direct CLI calls do not acquire a lock and are not serialized with CI. | ||
|
|
||
| `shared plan` writes the seven-case manifest without invoking functions. `shared | ||
| run` writes under `lmi_tests/artifacts/shared/standard/`: the manifest, coverage, | ||
| JSON/JUnit reports, launch journal and retained case evidence. Each invocation | ||
| uses a case-owned control/evidence prefix derived from `runId` and `caseId`. | ||
| The case hash uses only `caseId` within the run prefix. Plans, payloads and evidence | ||
| have no additional cohort identifier. Generate new run artifacts with this pinned | ||
| revision; inspect or clean up older artifacts with their original adapter revision. A service resume is journaled under | ||
| the same durable execution; it is not another driver Invoke. | ||
|
|
||
| Trusted same-repository pull requests and manual workflow runs exercise the cloud | ||
| suite. Fork and Dependabot pull requests run only the unprivileged local harness. | ||
| There is no S3 lease, fence token or persistent quarantine record. Existing S3 | ||
| ownership objects are not read and cannot block a later workflow run. | ||
|
|
||
| ## Actual observations and failure handling | ||
|
|
||
| `fixture.py` uses public durable APIs for the scenarios. Its instrumentation | ||
| observer captures actual operation IDs, parents, stored outcomes and user-task | ||
| lifetimes. The service-client wrapper forwards every SDK request and response; | ||
| it never supplies synthetic checkpoints or changes SDK cancellation flags. | ||
| The instrumentation observer is a measurement tool, not a plugin conformance test. | ||
|
|
||
| For normal return/failure, the adapter records the actual durable wrapper boundary | ||
| and probes the original context after that boundary. The operation probe uses | ||
| `DurableContext.step`; the checkpoint probe calls the SDK's | ||
| `context.state.create_checkpoint(is_sync=False)` admission path without issuing a | ||
| raw Lambda request. SDK closure/revocation errors are recorded as rejection only when no user code | ||
| or new SDK request ran. Probes run after both wrapper return and wrapper raise. An accepted checkpoint, another error, or a still-pending operation | ||
| remains visible. Probe tasks are tracked separately; a timeout cannot make them | ||
| quiescent. Current SDK gaps are failures, not expected passes or exclusions. | ||
|
|
||
| The deadline fixture holds real I/O across the runtime deadline and cooperatively | ||
| exits after it. A runtime clock alone never supplies a `platform_timeout` event: | ||
| that event requires the original request's service `InvocationCompleted` history. | ||
| The recovery victim first waits in a checkpointed preparation step. The driver | ||
| starts the peer halfway through the victim's native timeout, verifies the original | ||
| cohort and the peer's remaining recovery window, then releases preparation to | ||
| admit the fault. Preparation checkpoints the original native request ID, so | ||
| service retries keep the same durable sequence without reinjecting the victim's | ||
| fault or admission probes. Recovery overlap excludes expired original requests | ||
| and their retries. The original peer remains held until recovery completes. A still-running | ||
| durable execution is collected as such; terminal-state waiting and stops belong | ||
| to retirement after assertions. Releasing a victim fault cannot establish the | ||
| preceding cleanup or recovery assertion. | ||
|
|
||
| The warm fixture creates invocation-owned late work inside a step. A later | ||
| invocation must find it in the same process; it observes the old callback using | ||
| the original request identity. Failure to reuse that process is a placement or | ||
| collection failure, not permission to select a replacement victim. | ||
|
|
||
| `shared_evidence.py` supplies Python error expectations and process identity to | ||
| the shared source-evidence normalizer. | ||
| Service payload wrappers are decoded consistently with checkpoint snapshots; | ||
| truncated results are rejected. Replay compares a real resume's initial | ||
| checkpoints to the final service history and counts actual user-function entries. | ||
| Missing observations are never filled from expected answers. | ||
| Each native request writes a final protocol 2 seal after its wrapper and | ||
| registered work finish and all captured observations reach S3. A warm owner | ||
| can seal after its wrapper returns, while its callback finishes in the reused | ||
| process. Lost records, live probes, and a publisher killed at the native | ||
| deadline leave the request unsealed. If work still publishes after a seal, | ||
| its later sequence invalidates the seal when the driver reads it. | ||
|
|
||
| The fixture stops pre-seal publication once retry and read-back work reaches | ||
| five cumulative seconds, after the in-flight write and its read-back finish. | ||
| Successful first-attempt uploads do not use that budget. Slow successful | ||
| uploads can still consume the 60-second native invocation window. Exhausting | ||
| the retry budget leaves collection incomplete without replacing the SDK result. | ||
| Late writes after a seal still run so they can invalidate it. The published | ||
| workflow pin does not enforce terminal seals. Seal enforcement is tested | ||
| against the paired runner. The published pin also uses its original per-operation | ||
| replay counts. Logical map inputs and late-only replay effects require the | ||
| paired runner's policy support. | ||
|
|
||
| The shared behavior schema asks for one scenario-owned `task_exit`. Other user | ||
| and probe tasks remain in the same evidence as `observed_task_start/exit`, with | ||
| their original `sourceKind`. The independent retirement snapshot restores all | ||
| original task kinds, so a pending task cannot disappear from the audit. | ||
| Coordination barriers retain their complete `controlRecords` journal separately | ||
| from injected victim `faultsCreated`; releasing a healthy peer is not victim | ||
| fault recovery. | ||
|
|
||
| Run `hatch run lmi:shared-cleanup` for manual cleanup of the case manifests. | ||
| With the published pin, pass the same `--output` to run and cleanup when | ||
| using a custom case output directory. | ||
| When recovering an interrupted deployment with a journal-aware conformance | ||
| revision, set `LMI_SHARED_RUN_ID` to the run ID recorded in | ||
| `lmi_tests/artifacts/deployment-update.json`. Keep the journal in its original | ||
| artifact directory until reconciliation completes. | ||
| Cleanup releases only the current run's controls and requests stops only for its | ||
| identified executions. It validates run, case and function identity before those | ||
| requests. It records `NO_CASES`, `CLEANUP_REQUESTED` or `FAILED` in `cleanup.json`; | ||
| completed cleanup requests do not claim that every task retired. Assertion and | ||
| retirement failures remain in the shared reports. The shared runner can prevent | ||
| reuse of an unretired target within the current run, but that state is not stored | ||
| as a lock for future workflow runs. |
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| """Python fixture binding to the shared AWS driver.""" | ||
|
|
||
| from aws_durable_execution_conformance_tests.lmi.cloud import Cloud as SharedCloud | ||
| from lmi_tests.deploy import ARTIFACTS, manager | ||
|
|
||
|
|
||
| class Cloud(SharedCloud): | ||
| def __init__( | ||
| self, manifest, configuration=None, *, artifacts_dir=None, case_context=None | ||
| ): | ||
| super().__init__( | ||
| manifest, | ||
| configuration, | ||
| artifacts_dir=artifacts_dir or ARTIFACTS, | ||
| case_context=case_context, | ||
| verify_profile=manager().verify, | ||
| ) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.