Repository navigation
Conversation
Replace the checked-in taxi table (Parquet files tracked with Git LFS) with tables generated at test time by a new unpublished crate, iceberg-datafusion-test-tools. The crate provides a FixtureCatalog, an in-memory catalog over a temp dir, in which a TableBuilder creates tables and each FixtureTable::append commits one snapshot whose data files are exactly the batches given. Data files are written with iceberg-rust's writers, so they carry real column statistics. Snapshots are committed by the crate itself so that snapshot ids count up from 1, which keeps metadata-table snapshots deterministic. create_taxi_table generates a synthetic stand-in for the old fixture with the same schema and layout: 7 days x 25,000 rows, identity partitioned on pickup_date, one row group per file, without page indexes. Rows come from rand_pcg seeded per day, so they're the same on every run and platform. IcebergTestHarness moves from the published crate's test_utils module to crates/datafusion/tests/common, which drops the serde and typetag dependencies from iceberg-datafusion. Snapshot values change with the new data; plans don't. Git LFS is no longer needed by CI. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
|
im a big fan of generating and not committing test files to the repo 😄 |
gabotechs
marked this pull request as ready for review
October 26, 2026 17:06
The integration tests already exercise the taxi preset, so drop its tests in test-tools, except for one property: page_index.rs relies on the taxi files having no page indexes but returns the same rows either way, so it couldn't notice if they gained some. Check that in page_index.rs itself, next to the test that depends on it. The append tests stay: no integration test commits twice or passes a batch whose columns differ from the table's, so they're the only coverage of carrying earlier files forward and of matching columns. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The checked-in taxi table can only model one case: a single-snapshot V2 table with seven files. PRs that needed anything else built it by hand. #67 hand-writes manifests to get a position delete, #55 hand-writes sorted files and rewrites metadata in place, and #65 couldn't test page-index pruning at all because the taxi files have no page indexes.
This replaces the committed fixture with tables generated at test time, by a new unpublished crate,
iceberg-datafusion-test-tools(crates/test-tools):FixtureCatalogis an in-memory catalog over a temp dir.fixtures.table(ident, schema).partition_by(..).property(..).create()creates a table.table.append().writer_properties(..).file(batch)...commit()commits one snapshot in which every batch becomes exactly one data file. Columns are matched by name and cast to the table's types.fast_appendpicks random ids and can't commit delete files, so this is also where deletes can be added later.create_taxi_tablegenerates a synthetic stand-in for the old fixture with the same schema and layout: 7 days × 25,000 rows, identity-partitioned onpickup_date, one file per day, each a single row group with column chunk statistics but no page indexes. Rows come fromrand_pcg, seeded per day, so they're the same on every run and platform.randandrand_pcgwere already in the lockfile.IcebergTestHarnessmoves from the published crate'stest_utilsmodule tocrates/datafusion/tests/common, which also dropsserdeandtypetagfromiceberg-datafusion's dependencies. Snapshot values change with the new data; plans don't. Git LFS is no longer needed, so the CI checkout no longer fetches it.Generating the table takes about 0.9 s per harness in debug builds and 0.1 s in release, almost all of it Parquet encoding.
🤖 Generated with Claude Code