test: generate the taxi fixture instead of committing it

.github/workflows/ci.yml

@@ -76,7 +76,6 @@ jobs:
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
-          lfs: true
           persist-credentials: false
       - name: Setup Rust
         uses: ./.github/actions/setup-rust

AGENTS.md

@@ -26,6 +26,7 @@ Apache DataFusion integration for Apache Iceberg: catalog/table providers and ph
 | `crates/datafusion` | `iceberg-datafusion` | The published library |
 | `crates/sqllogictest` | `iceberg-sqllogictest` | SQL-level test harness (unpublished) |
 | `crates/playground` | `iceberg-playground` | REPL CLI built on `datafusion-cli` (unpublished) |
+| `crates/test-tools` | `iceberg-datafusion-test-tools` | Builders for the Iceberg tables tests read (unpublished) |
 
 ## Before Committing
 
@@ -52,7 +53,11 @@ cargo insta test -p iceberg-datafusion --test integration_datafusion_test --revi
 cargo run -p iceberg-playground -- --rc <catalogs.toml>                          # REPL; defaults to ~/.icebergrc
 ```
 
-Tests are self-contained (in-memory catalogs, temp dirs, checked-in metadata JSON); no Docker or external services are needed. The playground config holds `[[catalogs]]` entries with `name`, `type` (`rest` or `memory`), and a `[catalogs.config]` table of string properties.
+Tests are self-contained (in-memory catalogs, temp dirs, checked-in metadata JSON); no Docker or external services are needed.
+
+Build test tables with `iceberg-datafusion-test-tools` rather than by hand or by committing data files. A `FixtureCatalog` is an in-memory catalog over a temp dir, not tied to any dataset. `fixtures.table(ident, schema).partition_by(..).create()` creates a table, and each `table.append().file(batch)...commit()` adds one snapshot in which every batch becomes exactly one data file. Snapshot ids count up from 1, so snapshots can show them. Data files are written with iceberg-rust's writers, not this crate's insert path, so they carry real column statistics and a writer bug can't hide a reader bug. `create_taxi_table` generates a deterministic, synthetic stand-in for NYC taxi trips: 7 days × 25,000 rows, identity-partitioned on `pickup_date`, one file per day.
+
+The integration tests share `IcebergTestHarness` from `crates/datafusion/tests/common/`. It's a session with `datafusion.public.taxi` registered, so `FROM taxi` resolves unqualified. The playground config holds `[[catalogs]]` entries with `name`, `type` (`rest` or `memory`), and a `[catalogs.config]` table of string properties.
 
 ## Architecture (`crates/datafusion`)
 

Cargo.toml

@@ -20,6 +20,7 @@ members = [
     "crates/datafusion",
     "crates/sqllogictest",
     "crates/playground",
+    "crates/test-tools",
 ]
 resolver = "2"
 
@@ -36,6 +37,7 @@ async-trait = "0.1.89"
 datafusion = "55.0.0"
 datafusion-cli = "55.0.0"
 iceberg-datafusion = { path = "crates/datafusion" }
+iceberg-datafusion-test-tools = { path = "crates/test-tools" }
 datafusion-sqllogictest = "55.0.0"
 iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "5b82ab99aeaf50bf4f5b3b776ac9cd6380a5f3e3" }
 iceberg-catalog-rest = { git = "https://github.com/apache/iceberg-rust", rev = "5b82ab99aeaf50bf4f5b3b776ac9cd6380a5f3e3" }

README.md

@@ -30,6 +30,8 @@ through an Iceberg catalog.
 - [`crates/sqllogictest`](crates/sqllogictest): SQL logic tests for the integration.
 - [`crates/playground`](crates/playground): a command-line SQL playground backed
   by DataFusion and Iceberg catalogs.
+- [`crates/test-tools`](crates/test-tools): builders for the Iceberg tables the
+  tests read.
 
 ## Query an existing table
 

crates/datafusion/Cargo.toml

@@ -38,10 +38,9 @@ iceberg = { workspace = true }
 parquet = "59.2"
 tokio = { workspace = true }
 uuid = { version = "1.18", features = ["v7"] }
-serde = { version = "1", features = ["derive"] }
-typetag = "0.2"
 
 [dev-dependencies]
+iceberg-datafusion-test-tools = { workspace = true }
 insta = "1"
 parquet = "59.2"
 tempfile = "3.18"

crates/datafusion/src/lib.rs

@@ -28,6 +28,3 @@ pub use table_provider::{
     IcebergCatalogTableProvider, IcebergStaticTableProvider, IcebergTableProviderFactory,
 };
 pub mod write;
-
-#[doc(hidden)]
-pub mod test_utils;