Repository navigation
Conversation
ewoodbury
reviewed
Oct 28, 2026
ewoodbury
left a comment
Contributor
There was a problem hiding this comment.
Contributing my review here, changes look solid. Agree with the approach of promoting the bound where it is compared with a checked multiply, and it follows #3483. I confirmed the new evaluator tests fail with the date arms reverted, so they pin the fix.
Main asks from me before merge:
- Handle the spec's 4-byte bound case under a timestamp/timestamp_ns schema. Those manifests still fail to load.
- Settle whether the partition-value half belongs here. My reading of the spec is that it rules out this promotion for identity partitions.
- Pin the pruned side of <, > and IN.
Date columns promoted to timestamp or timestamp_ns keep day-count lower/upper bounds in older manifests. Convert those bounds to midnight of each day (including 4-byte date payloads under a timestamp schema) so bound pruning keeps working across the promotion. Dates that overflow the target range return DataInvalid. Partition date→timestamp is not promoted: the spec rules that out for identity transforms. Closes apache#3503
ump45nose
force-pushed
the
fix/3503-date-bound-promotion
branch
from
October 28, 2026 13:34
84dd5ba to
8fc8a81
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
Follow-up to #3483. With v3 type promotion a
datecolumn can becometimestamportimestamp_ns, but older manifests keep day-count lower/upper bounds. Compared as raw numbers those day counts sit far below any timestamp in micros/nanos, so bound pruning wrongly drops files.Datum::promote_toconverts a date (days from epoch) totimestamp(micros) ortimestamp_ns(nanos) via checked multiplication; a date outside the target range returnsDataInvalid. Matching is on(self.data_type(), target)so a bareIntis not treated as a date.Datum::try_from_bytesaccepts 4-byte payloads under atimestamp/timestamp_nsschema (manifests that carried an older date bound forward) and reuses the same conversion.InclusiveMetricsEvaluatorpromotes date lower/upper bounds before comparing them to a timestamp predicate (field name attached on error) and propagates overflow asDataInvalid.Partition
date→timestamp/timestamp_nsis not promoted: the spec rules that out for identity transforms.timestamptztargets are left alone (date → timestamptz is not an allowed promotion).Are these changes tested?
Yes, new / folded unit tests:
spec::values::tests::test_datum_promote_date: table-driven date → timestamp / timestamp_ns (including negative),timestamp_nsedges day 106751 fits / 106752 doesn't, overflow →DataInvalid, bareIntdoes not promote.spec::values::tests::avro_bytes_timestamp_from_dateandtest_parse_manifest_promotes_date_bounds_under_timestamp_schema: 4-byte date bounds under a timestamp schema.inclusive_metrics_evaluator::eval_promotes_date_bounds: table-driven</>/=/INwith inside (kept) and outside (pruned) literals for both targets; overflow assertsErrorKind::DataInvalid.AI Disclosure
I used an AI coding assistant to help write the code and tests. I reviewed the changes and ran the tests locally.