Summary
Add Stage 3 of the canonical US build-stage work: live runtime manifest writing for the versioned US Microplex build path.
The goal is to make the package update stage manifests as the pipeline runs, instead of only reconstructing stage output manifests after a saved artifact bundle exists. This should make partial runs diagnosable and prepare the package for future conditional/resumable builds without enabling automatic conditional execution yet.
Intended behavior
- Initialize a runtime stage writer before source loading in the canonical multi-source versioned build entrypoint.
- Write Stage 1 immediately from run/config/provider metadata.
- Update Stage 2 as source providers load frames.
- Record stage starts, outputs, diagnostics, completion, deferral, and failures for Stages 1-9.
- Preserve failed/deferred/completed lifecycle state when finalizing manifests from the saved artifact manifest.
- Publish stable stage artifacts at the existing boundaries, including Stage 4 scaffold seed, Stage 6 pre-calibration PolicyEngine entity tables, Stage 7 calibrated entity tables, Stage 8 dataset assembly artifacts, and Stage 9 validation evidence.
- Keep Stage 9 replay explicit and non-mutating against existing Stage 8 datasets.
- Add seam tests that ensure each stage output manifest can be consumed by the next stage contract.
Out of scope
- Automatic conditional execution or skipping earlier build stages.
- Running a full production nationwide dataset build locally.
- Replacing the existing source/provider/modeling implementation.
Local verification
Because this machine is macOS x86_64, uv run attempted to install torch==2.11.0, which has no compatible macOS x86_64 wheel. The verification was therefore run through the existing local .venv rather than forcing dependency resolution.
Passing checks:
.venv/bin/python -m pytest tests/pipelines/test_versioned_artifacts.py::test_build_and_save_versioned_us_microplex_from_source_providers -q.venv/bin/python -m pytest tests/pipelines/test_stage_runtime.py tests/pipelines/test_stage_run.py tests/pipelines/test_stage_artifacts.py tests/pipelines/test_stage_contracts.py -q.venv/bin/python -m pytest tests/pipelines/test_stage9_replay.py -q
Summary
Add Stage 3 of the canonical US build-stage work: live runtime manifest writing for the versioned US Microplex build path.
The goal is to make the package update stage manifests as the pipeline runs, instead of only reconstructing stage output manifests after a saved artifact bundle exists. This should make partial runs diagnosable and prepare the package for future conditional/resumable builds without enabling automatic conditional execution yet.
Intended behavior
Out of scope
Local verification
Because this machine is macOS x86_64,
uv runattempted to installtorch==2.11.0, which has no compatible macOS x86_64 wheel. The verification was therefore run through the existing local.venvrather than forcing dependency resolution.Passing checks:
.venv/bin/python -m pytest tests/pipelines/test_versioned_artifacts.py::test_build_and_save_versioned_us_microplex_from_source_providers -q.venv/bin/python -m pytest tests/pipelines/test_stage_runtime.py tests/pipelines/test_stage_run.py tests/pipelines/test_stage_artifacts.py tests/pipelines/test_stage_contracts.py -q.venv/bin/python -m pytest tests/pipelines/test_stage9_replay.py -q