Skip to content

fix(scan): promote date column bounds to timestamp - #3522

Open
ump45nose wants to merge 1 commit into
apache:mainfrom
ump45nose:fix/3503-date-bound-promotion
Open

ump45nose wants to merge 1 commit into
apache:mainfrom
ump45nose:fix/3503-date-bound-promotion

Conversation

@ump45nose

@ump45nose ump45nose commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

Follow-up to #3483. With v3 type promotion a date column can become timestamp or timestamp_ns, but older manifests keep day-count lower/upper bounds. Compared as raw numbers those day counts sit far below any timestamp in micros/nanos, so bound pruning wrongly drops files.

  • Datum::promote_to converts a date (days from epoch) to timestamp (micros) or timestamp_ns (nanos) via checked multiplication; a date outside the target range returns DataInvalid. Matching is on (self.data_type(), target) so a bare Int is not treated as a date.
  • Datum::try_from_bytes accepts 4-byte payloads under a timestamp / timestamp_ns schema (manifests that carried an older date bound forward) and reuses the same conversion.
  • InclusiveMetricsEvaluator promotes date lower/upper bounds before comparing them to a timestamp predicate (field name attached on error) and propagates overflow as DataInvalid.

Partition date → timestamp / timestamp_ns is not promoted: the spec rules that out for identity transforms.

timestamptz targets are left alone (date → timestamptz is not an allowed promotion).

Are these changes tested?

Yes, new / folded unit tests:

  • spec::values::tests::test_datum_promote_date: table-driven date → timestamp / timestamp_ns (including negative), timestamp_ns edges day 106751 fits / 106752 doesn't, overflow → DataInvalid, bare Int does not promote.
  • spec::values::tests::avro_bytes_timestamp_from_date and test_parse_manifest_promotes_date_bounds_under_timestamp_schema: 4-byte date bounds under a timestamp schema.
  • inclusive_metrics_evaluator::eval_promotes_date_bounds: table-driven < / > / = / IN with inside (kept) and outside (pruned) literals for both targets; overflow asserts ErrorKind::DataInvalid.

AI Disclosure

I used an AI coding assistant to help write the code and tests. I reviewed the changes and ran the tests locally.

@ewoodbury ewoodbury left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contributing my review here, changes look solid. Agree with the approach of promoting the bound where it is compared with a checked multiply, and it follows #3483. I confirmed the new evaluator tests fail with the date arms reverted, so they pin the fix.

Main asks from me before merge:

  • Handle the spec's 4-byte bound case under a timestamp/timestamp_ns schema. Those manifests still fail to load.
  • Settle whether the partition-value half belongs here. My reading of the spec is that it rules out this promotion for identity partitions.
  • Pin the pruned side of <, > and IN.

Comment thread crates/iceberg/src/spec/values/primitive.rs Outdated
Comment thread crates/iceberg/src/scan/context.rs Outdated
Comment thread crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs Outdated
Comment thread crates/iceberg/src/spec/values/datum.rs
Comment thread crates/iceberg/src/spec/values/primitive.rs Outdated
Date columns promoted to timestamp or timestamp_ns keep day-count
lower/upper bounds in older manifests. Convert those bounds to midnight
of each day (including 4-byte date payloads under a timestamp schema) so
bound pruning keeps working across the promotion. Dates that overflow
the target range return DataInvalid.

Partition date→timestamp is not promoted: the spec rules that out for
identity transforms.

Closes apache#3503
@ump45nose
ump45nose force-pushed the fix/3503-date-bound-promotion branch from 84dd5ba to 8fc8a81 Compare October 28, 2026 13:34
@ump45nose ump45nose changed the title fix(scan): promote date partition values and column bounds to timestamp fix(scan): promote date column bounds to timestamp Oct 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reader does not promote date partition values and column bounds to timestamp / timestamp_ns

2 participants