Skip to content

PE-US rebuild smoke reaches post-microsim but microcalibrate requires real torch #175

@anth-volk

Description

@anth-volk

A checkpoint resume from post_imputation successfully materialized PE variables and saved a post_microsim checkpoint, then failed when the microcalibrate backend attempted to use torch tensor operations in an environment where torch had been stubbed only to unblock non-neural imports.

Resume command shape:

.venv/bin/python -m microplex_us.pipelines.pe_us_recalibrate_from_checkpoint \
  --checkpoint-path artifacts/local_us_microplex_smoke/local-smoke-v1/checkpoints/post_imputation \
  --output-root artifacts/local_us_microplex_smoke/local-smoke-v1/recalibrated_from_post_imputation \
  --targets-db /Users/administrator/Documents/PolicyEngine/policyengine-us-data/policyengine_us_data/storage/calibration/policy_data.db \
  --target-period 2024 \
  --calibration-backend microcalibrate \
  --policyengine-materialize-batch-size 100000 \
  --pipeline-checkpoint-save-post-microsim-path artifacts/local_us_microplex_smoke/local-smoke-v1/checkpoints/post_microsim

Progress before failure:

US microplex build: post-microsim checkpoint saved [path=artifacts/local_us_microplex_smoke/local-smoke-v1/checkpoints/post_microsim]

Failure:

  File ".venv/lib/python3.14/site-packages/microcalibrate/calibration.py", line 117, in __init__
    torch.manual_seed(self.seed)
  File ".venv/lib/python3.14/site-packages/torch/__init__.py", line 58, in _unavailable
    raise RuntimeError(...)
RuntimeError: The local smoke environment is using a torch stub. Install a real torch wheel before using neural synthesis or tensor operations.

The torch stub was used only because the project selected Python 3.14 and uv sync --extra policyengine could not install a real torch wheel on this workstation. The canonical smoke command uses --calibration-backend microcalibrate, so the environment still needs a real torch install even though neural synthesis is not used.

Next retry can resume from artifacts/local_us_microplex_smoke/local-smoke-v1/checkpoints/post_microsim, skipping donor integration and PE materialization.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions