Repository navigation
Remove the data_sample model constructor argument - #1082
Conversation
Hyrax now constructs every model as Model(config) and passes nothing else. The pre-flight layer that existed only to feed a data_sample is deleted: DataProvider.sample_data, StreamingDataProvider.sample_data, HyraxCSVDataset.sample_data, the Kafka and LSDB peek_sample methods with their replay buffers, the peek_timeout config key, setup_model_from_sample, and the reset_trace call that cleaned up after the pre-flight. The built-in image models (HyraxCNN, HyraxAutoencoder, HyraxAutoencoderV2, ImageDCAE) size their layers from [model.<Name>].input_shape, a [channels, height, width] list defaulting to [3, 32, 32]. A shared read_input_shape helper validates it and raises a ValueError naming the key. Height and width are now assigned correctly where they were swapped. HyraxTs2Vec sizes its input from EncodingSettings, and SimCLR is constructable through setup_model again. infer_stream no longer takes sample_batch: a configured [data_request.infer_stream] group gives an iterable session, otherwise a manual session fed through process(). Migration 007 drops peek_timeout from existing configs. Docs, the model reference, and the maintained notebooks are updated to the new contract. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1082 +/- ##
==========================================
+ Coverage 71.02% 71.56% +0.53%
==========================================
Files 92 93 +1
Lines 9381 9309 -72
==========================================
- Hits 6663 6662 -1
+ Misses 2718 2647 -71 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
drewoldag
left a comment
There was a problem hiding this comment.
This is looking pretty correct to me. One small update to the way a config value is being accessed, and a comment about docstring in infer_stream (not a blocker)
| 2. **Manual** — leave ``[data_request.infer_stream]`` unset and feed batches | ||
| yourself:: |
There was a problem hiding this comment.
This seems odd to me. The original reasoning for having this functionality isn't clear at this point either. I don't think there's any code to remove. I think just removing the documentation would suffice.
Change Description
Every
@hyrax_modelclass was constructed asModel(config=config, data_sample=prepared_sample), and feeding that sample required a pre-flight layer:DataProvider.sample_data,StreamingDataProvider.sample_data,HyraxCSVDataset.sample_data, the Kafka and LSDBpeek_samplemethods with replay buffers and apeek_timeoutkey, andsetup_model_from_sampleforinfer_stream. Only five of nine built-in models read the sample, one of those could already get the value from config, andSimCLRcould not be constructed throughsetup_modelat all.This PR removes
data_sampleoutright, with no compatibility shim, and deletes the pre-flight machinery with it.Solution Description
Model contract. Hyrax constructs a model as
Model(config)and passes nothing else.setup_model(config)takes no dataset, andsetup_model_from_sampleandreset_trace(which only existed to clean up after the pre-flight) are gone.Image models read
[model.<Name>].input_shape.HyraxCNN,HyraxAutoencoder,HyraxAutoencoderV2, andImageDCAEsize their layers from a[channels, height, width]list, defaulting to[3, 32, 32]to matchHyraxCifarDataset. A sharedread_input_shapehelper inmodel_utilsvalidates it (three positive ints,falserejected) and raises aValueErrornaming the key. Users of other datasets set it alongside[data_set].crop_to. While here, the swapped width/height assignment in the autoencoders and CNN is fixed, which only changes behavior for non-square inputs.Other models.
HyraxTs2Vecsizes its input fromEncodingSettings(config).n_channels, which it already computed.HSCAutoencoder,HSCDCAE,HyraxLoopback, andSimCLRjust drop the parameter.infer_stream.run()no longer takessample_batch. A configured[data_request.infer_stream]group yields an iterable session; otherwise a manual session that you feed throughsession.process(batch).Peek machinery deleted. Kafka loses
peek_sample,PEEK_WARN_INTERVAL,peek_timeout, and_buffered. LSDB losespeek_sampleand_peek_indexbut keeps_buffered, which__iter__still uses to push back undelivered rows. Migration 007 dropspeek_timeoutfrom existing configs.Docs.
model_class_reference.rst,HYRAX_GUIDE.md, and the maintained notebooks follow the new contract.external_model_class.ipynbandlsdb_stream_dataset.ipynbnow read shapes from a config entry or a module constant. Archived notebooks are left as history.Testing
python -m pytest tests/ -m "not slow": 810 passed.python -m pytest tests/hyrax/test_e2e.py: 6 passed. The hsc1k demo data has three filters and cutouts of at least 96x96, so HSC and FITS run withcrop_to = [96, 96]andinput_shape = [3, 96, 96]; CIFAR runs on the default.tests/hyrax/test_input_shape.pychecks that each image model builds from the shipped default and that malformedinput_shapevalues (wrong length, non-int, zero, negative, bool,false, scalar) raise aValueErrornaming the key.infer_streamtests callh.infer_stream()with no argument.pre-commit run --all-filespasses with the sphinx hook skipped.Code Quality
🤖 Generated with Claude Code