[drewoldag]

Summary

The v0.13 performance work was driven by the self-hosted CIFAR-10 CNN benchmark, which only covers one model/dataset path. This adds a second runtime benchmark for the astronomy path Hyrax is built for: HyraxAutoencoderV2 + HSCDataset on the ~1000 HSC1k Zenodo cutouts (g/r/i FITS), timing training and inference against a pure-PyTorch equivalent.

• benchmarks_self_hosted/timebench_hsc_utils.py (new): mirrors timebench_utils.py.
  • HSCTorchDataset: a plain torch.utils.data.Dataset that reproduces HSCDataset's tensors (same file regex, sorted filters, min-square center crop, tanh). It reads FITS on every __getitem__ with no caching, i.e. idiomatic PyTorch.
  • AutoencoderV2: a standalone copy of the HyraxAutoencoderV2 architecture with no Hyrax imports.
  • check_architecture_parity: an untimed guard that compares state_dict keys/shapes against HyraxAutoencoderV2 built from default config, so the two can't silently drift apart.
  • benchmark_pytorch_hsc / benchmark_hyrax_hsc / benchmark_repeated_hsc: time train and infer; the quality metric is the final epoch-averaged training MSE (computed sample-weighted like _attach_epoch_metric_averaging; the Hyrax value is read from MLflow's training/epoch/avg_loss).
• run_timebench.py: adds --benchmark {cifar,hsc,all} (default cifar, so existing behavior is unchanged). Result files are now <commit>_<benchmark>_<timestamp>.json with a benchmark key; CIFAR keys are otherwise unchanged.
• benchmarks_self_hosted/README.md (new): how to run, env vars (CIFAR_DIR, HSC_DIR, RESULT_PATH), what each benchmark compares.
• .github/workflows/pipeline_time_bench.yml: exports HSC_DIR, runs --benchmark all, and copies every RESULT_FILE= line into gondor_results/.

Design decisions

• The comparison is "idiomatic PyTorch vs idiomatic Hyrax": the PyTorch loader does not cache, Hyrax keeps its default data_set.use_cache = true. use_cache is recorded per framework in the result JSON.
• Train on all data, infer on all data (split = {"train": 1.0}).
• Unlike the CIFAR baseline, the PyTorch side saves its trained weights and reloads them for inference, since Hyrax does the same.

Verification

• Scratch parity check: both datasets yield the same 993 objects; image tensors for sampled object ids match to within float32 rounding; model state_dicts match.
• Smoke runs with reduced configs (MPS): at 3 epochs the final MSE agrees (0.0544 PyTorch vs 0.0545 Hyrax). Hyrax trains faster here because of its cache, as intended.
• python benchmarks_self_hosted/run_timebench.py with no flag still runs CIFAR only.
• ruff and all pre-commit hooks pass except the Sphinx build, which fails on pre-existing docs warnings.

🤖 Generated with Claude Code

[drewoldag]

Don't have to change this in this PR, but clearly we don't want this to be the place where we store data.

[Copilot]

🟡 Changes recommended

Different sampling regimes and inconsistent handling of non-default training fractions undermine the framework comparisons.

2 open issues

• Match Hyrax replacement sampling in PyTorch training · New
• Reject train fractions that PyTorch does not apply · New

What changed in this PR

Adds an HSC1k autoencoder runtime benchmark alongside CIFAR-10, extending Hyrax’s performance comparisons to its astronomy workflow.

Changes:

• Adds PyTorch and Hyrax training/inference benchmarks with architecture checks and reconstruction-MSE reporting.
• Adds benchmark selection and separate, labeled JSON results while preserving CIFAR as the default.
• Documents usage and updates the self-hosted workflow to run and collect both benchmarks.

File	Description
benchmarks_self_hosted/​timebench_hsc_utils.py	Implements HSC data loading, autoencoder models, and repeated timing comparisons.
benchmarks_self_hosted/​run_timebench.py	Adds benchmark selection and per-benchmark result files.
benchmarks_self_hosted/​README.md	Documents benchmark behavior, commands, and environment variables.
.github/​workflows/​pipeline_time_bench.yml	Runs both benchmarks and publishes their result files.

_{Review effort: Balanced}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.56%. Comparing base (ff371ce) to head (5d7b510).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1080   +/-   ##
=======================================
  Coverage   71.56%   71.56%           
=======================================
  Files          93       93           
  Lines        9309     9309           
=======================================
  Hits         6662     6662           
  Misses       2647     2647           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[drewoldag]

AE+HSC1000 benchmark is looking pretty good!

{
  "benchmark": "hsc",
  "device": "mps",
  "train_fraction": 1.0,
  "epochs": 5,
  "batch_size": 128,
  "num_workers": 0,
  "lr": 0.01,
  "repeats": 3,
  "warmup": 1,
  "hyrax_train_time": 4.948955514313032,
  "pytorch_train_time": 11.623738249919066,
  "train_slowdown": 0.42576281467345417,
  "hyrax_infer_time": 3.097659472065667,
  "pytorch_infer_time": 3.1331079023269317,
  "infer_slowdown": 0.9886858571851492
}