Repository navigation
deps: bump arrow and parquet to 59.3.2 - #7419
Merged
Merged
Conversation
arrow-rs 59.3.2 includes apache/arrow-rs#11754, the 59.x backport of apache/arrow-rs#11021: DeltaBitPackDecoder::skip reads miniblocks wider than 64 values instead of failing with "cannot skip miniblock of size N". Snowflake writes 512-value miniblocks and Databricks Photon 256, and a native scan reaches skip when it skips a long run of rows, such as a run of Iceberg position deletes. The native and Delta contrib lockfiles move arrow-*, parquet and parquet-variant* from 59.3.1 to 59.3.2. See apache#7417.
github-actions
Bot
added
dependencies
Member
Author
|
@andygrove could this be backported to branch-1.1 and branch-1.2? |
andygrove
reviewed
Nov 2, 2026
| dependencies = [ | ||
| "heck 0.5.0", | ||
| "itertools 0.14.0", | ||
| "itertools 0.13.0", |
Member
Author
There was a problem hiding this comment.
Not intended, fixed
andygrove
reviewed
Nov 2, 2026
| dependencies = [ | ||
| "anyhow", | ||
| "itertools 0.14.0", | ||
| "itertools 0.13.0", |
andygrove
added
backport-1.1
cargo update re-picked itertools 0.13.0 for them when moving arrow and parquet. Restore the previous choice, so the lockfiles only change the arrow and parquet crates.
Member
Author
|
@andygrove Thank you, good catch, fixed |
fresh-borzoni
force-pushed
the
deps-parquet-59.3.2
branch
from
November 2, 2026 21:44
ffb80d9 to
574481a
Compare
Member
Author
|
Fixed the RSS frame-size test: arrow-rs 59.3.2 emits an empty validity bitmap for non-null columns, so the frame is 2,048 bytes smaller. |
Member
@fresh-borzoni Sounds good. Do you want to create the back-port PRs? You (or your agent) can read the contributor guide to know the rules about this. |
Member
Author
|
@andygrove Thank you, sure, will open both once this merges 👍 |
This was referenced Nov 3, 2026
Member
Author
|
@andygrove Opened #7447 and #7448 for backports, thank you 🙏 |
andygrove
pushed a commit
that referenced
this pull request
Nov 3, 2026
* deps: bump arrow and parquet to 59.3.2 arrow-rs 59.3.2 includes apache/arrow-rs#11754, the 59.x backport of apache/arrow-rs#11021: DeltaBitPackDecoder::skip reads miniblocks wider than 64 values instead of failing with "cannot skip miniblock of size N". Snowflake writes 512-value miniblocks and Databricks Photon 256, and a native scan reaches skip when it skips a long run of rows, such as a run of Iceberg position deletes. The native and Delta contrib lockfiles move arrow-*, parquet and parquet-variant* from 59.3.0 to 59.3.2. See #7417. * deps: keep prost-build and prost-derive on itertools 0.14.0 cargo update re-picked itertools 0.13.0 for them when moving arrow and parquet. Restore the previous choice, so the lockfiles only change the arrow and parquet crates. * test: update the RSS frame size for arrow-rs 59.3.2 (cherry picked from commit 644a870)
andygrove
pushed a commit
that referenced
this pull request
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #7417.
Rationale for this change
Pick up the fixes in arrow-rs 59.3.2. It is a patch release over 59.3.1 with 15 backported fixes. The changes most relevant to Comet are:
DELTA_BINARY_PACKEDcolumns with miniblocks wider than 64 values, which failed withcannot skip miniblock of size N(Native Iceberg scan fails on DELTA_BINARY_PACKED miniblocks wider than 64 values #7417)page_encoding_statsBuffer::shrink_to_fitFull changelog: https://github.com/apache/arrow-rs/blob/59.3.2/CHANGELOG.md
What changes are included in this PR?
cargo update, moving thearrow-*,parquetandparquet-variant*crates from 59.3.1 to 59.3.2.DataFusion, iceberg-rust and object_store remain unchanged.
How are these changes tested?
Locally verified on macOS/aarch64:
cargo check --workspace --all-targets --lockedcargo clippy --workspace --all-targets --locked -- -D warningscargo fmt --all -- --checkcargo check -p datafusion-comet --features contrib-delta --lockedcargo check --locked --all-targetsincontrib/delta/nativeWith this change, a 1.2.0-rc1 build returns Spark's results for the reproduction in #7417.
The
run-iceberg-testsandrun-delta-build-gatelabels would request the corresponding CI coverage.