Skip to content

Remove the data_sample model constructor argument - #1082

Merged
drewoldag merged 3 commits into
mainfrom
awo/remove-data-sample
Oct 28, 2026
Merged

drewoldag merged 3 commits into
mainfrom
awo/remove-data-sample

Conversation

@drewoldag

Copy link
Copy Markdown
Collaborator

Change Description

Every @hyrax_model class was constructed as Model(config=config, data_sample=prepared_sample), and feeding that sample required a pre-flight layer: DataProvider.sample_data, StreamingDataProvider.sample_data, HyraxCSVDataset.sample_data, the Kafka and LSDB peek_sample methods with replay buffers and a peek_timeout key, and setup_model_from_sample for infer_stream. Only five of nine built-in models read the sample, one of those could already get the value from config, and SimCLR could not be constructed through setup_model at all.

This PR removes data_sample outright, with no compatibility shim, and deletes the pre-flight machinery with it.

Solution Description

Model contract. Hyrax constructs a model as Model(config) and passes nothing else. setup_model(config) takes no dataset, and setup_model_from_sample and reset_trace (which only existed to clean up after the pre-flight) are gone.

Image models read [model.<Name>].input_shape. HyraxCNN, HyraxAutoencoder, HyraxAutoencoderV2, and ImageDCAE size their layers from a [channels, height, width] list, defaulting to [3, 32, 32] to match HyraxCifarDataset. A shared read_input_shape helper in model_utils validates it (three positive ints, false rejected) and raises a ValueError naming the key. Users of other datasets set it alongside [data_set].crop_to. While here, the swapped width/height assignment in the autoencoders and CNN is fixed, which only changes behavior for non-square inputs.

Other models. HyraxTs2Vec sizes its input from EncodingSettings(config).n_channels, which it already computed. HSCAutoencoder, HSCDCAE, HyraxLoopback, and SimCLR just drop the parameter.

infer_stream. run() no longer takes sample_batch. A configured [data_request.infer_stream] group yields an iterable session; otherwise a manual session that you feed through session.process(batch).

Peek machinery deleted. Kafka loses peek_sample, PEEK_WARN_INTERVAL, peek_timeout, and _buffered. LSDB loses peek_sample and _peek_index but keeps _buffered, which __iter__ still uses to push back undelivered rows. Migration 007 drops peek_timeout from existing configs.

Docs. model_class_reference.rst, HYRAX_GUIDE.md, and the maintained notebooks follow the new contract. external_model_class.ipynb and lsdb_stream_dataset.ipynb now read shapes from a config entry or a module constant. Archived notebooks are left as history.

Testing

  • python -m pytest tests/ -m "not slow": 810 passed.
  • python -m pytest tests/hyrax/test_e2e.py: 6 passed. The hsc1k demo data has three filters and cutouts of at least 96x96, so HSC and FITS run with crop_to = [96, 96] and input_shape = [3, 96, 96]; CIFAR runs on the default.
  • New tests/hyrax/test_input_shape.py checks that each image model builds from the shipped default and that malformed input_shape values (wrong length, non-int, zero, negative, bool, false, scalar) raise a ValueError naming the key.
  • Peek tests removed; manual-mode infer_stream tests call h.infer_stream() with no argument.
  • pre-commit run --all-files passes with the sphinx hook skipped.

Code Quality

  • I have read the Contribution Guide and agree to the Code of Conduct
  • My code follows the code style of this project
  • My code builds (or compiles) cleanly without any errors or warnings
  • My code contains relevant comments and necessary documentation

🤖 Generated with Claude Code

Hyrax now constructs every model as Model(config) and passes nothing else.
The pre-flight layer that existed only to feed a data_sample is deleted:
DataProvider.sample_data, StreamingDataProvider.sample_data,
HyraxCSVDataset.sample_data, the Kafka and LSDB peek_sample methods with
their replay buffers, the peek_timeout config key, setup_model_from_sample,
and the reset_trace call that cleaned up after the pre-flight.

The built-in image models (HyraxCNN, HyraxAutoencoder, HyraxAutoencoderV2,
ImageDCAE) size their layers from [model.<Name>].input_shape, a
[channels, height, width] list defaulting to [3, 32, 32]. A shared
read_input_shape helper validates it and raises a ValueError naming the
key. Height and width are now assigned correctly where they were swapped.
HyraxTs2Vec sizes its input from EncodingSettings, and SimCLR is
constructable through setup_model again.

infer_stream no longer takes sample_batch: a configured
[data_request.infer_stream] group gives an iterable session, otherwise a
manual session fed through process().

Migration 007 drops peek_timeout from existing configs. Docs, the model
reference, and the maintained notebooks are updated to the new contract.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 19:11
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@drewoldag
drewoldag marked this pull request as draft October 28, 2026 19:12
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.27586% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 71.56%. Comparing base (dade46f) to head (4a80b47).

Files with missing lines Patch % Lines
src/hyrax/models/hyrax_autoencoder.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1082      +/-   ##
==========================================
+ Coverage   71.02%   71.56%   +0.53%     
==========================================
  Files          92       93       +1     
  Lines        9381     9309      -72     
==========================================
- Hits         6663     6662       -1     
+ Misses       2718     2647      -71     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread src/hyrax/trace.py Resolved

@drewoldag drewoldag left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty correct to me. One small update to the way a config value is being accessed, and a comment about docstring in infer_stream (not a blocker)

Comment thread src/hyrax/models/model_utils.py Resolved
Comment on lines +49 to +50
2. **Manual** — leave ``[data_request.infer_stream]`` unset and feed batches
yourself::

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems odd to me. The original reasoning for having this functionality isn't clear at this point either. I don't think there's any code to remove. I think just removing the documentation would suffice.

@drewoldag
drewoldag marked this pull request as ready for review October 28, 2026 21:15

@mtauraso mtauraso left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@drewoldag
drewoldag merged commit ff371ce into main Oct 28, 2026
8 checks passed
@drewoldag
drewoldag deleted the awo/remove-data-sample branch October 28, 2026 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants