perf(parquet): count fitting values before scanning def levels by adriangb · Pull Request #11997 · apache/arrow-rs

adriangb · 2026-10-30T00:55:42Z

Which issue does this PR close?

Related to parquet: clustered nulls can overflow dictionary pages despite byte-budget batching #11994 / fix(parquet): cap byte-budget level windows by value count #11996: this is a follow-up perf tweak, not the fix itself.

Rationale for this change

ByteBudgetChunker::byte_budget_sub_batch calls chunk_def.value_count(...) before it asks the encoder whether the chunk's values fit the page budget. For nullable columns (Materialized def levels) that call is an O(chunk_size) scan, and it runs on every chunk of every nullable byte-array column, even chunks far below the page limit.

What changes are included in this PR?

The chunker now asks the encoder first, about the next chunk_size values. That window always contains the chunk's values as a prefix, because a chunk holds at most chunk_size values.

count_values_within_byte_budget[_gather] counts how many leading values fit, so its answer for the window also answers for the chunk:

The whole window fits: the chunk fits too. Return None without scanning def levels. This is the common case, and it costs O(1) for offset and view arrays, using the existing stage-1 bounds.
The count stops at k, the first value that overflows: only now run value_count. If k >= vals_in_chunk the overflow is past this chunk, so the chunk fits. Otherwise k is the same answer the current code computes.

The lookahead is used only on the gather path (value_indices), which the arrow writer always uses, because every gather index names a real value. A contiguous values slice from a raw ColumnWriter caller may be padded with placeholder ByteArrays past the values its levels reference, and those can't be sized. That path keeps counting the chunk's values first, as on main.

There are no new trait methods. An earlier revision of this PR added a byte-array-only values_fit_within_byte_budget_gather hook to ColumnValueEncoder. This version reuses the existing count instead, and benchmarked the same.

Are these changes tested?

New unit tests in byte_budget_chunker call pick_sub_batch directly, through both the contiguous and the gather entry points:

a small chunk is not split;
an overflow past the chunk, inside the lookahead window, does not split it;
an overflow inside the chunk splits it, with the same Levels window as on main;
an all-null chunk is not split;
a contiguous slice padded with placeholder values past the chunk is never read (test_write_statistics_not_arrow in tests/geospatial.rs caught this in CI).

They pass on main as well, so they pin the behaviour to main's. The second test fails if the k >= vals_in_chunk check is removed. The existing writer and page-size tests also pass.

Benchmarks: arrow_writer, /default props, on a quiet GCE c3-standard-8. Each variant ran twice, interleaved, and the numbers below are the two runs:

case	main	this PR
byte_array_sparse_99pct_null	10.6 / 10.7 ms	9.3 / 9.3 ms (−13%)
string_and_binary_view	52.9 / 52.6 ms	51.7 / 51.5 ms
string	150.9 / 152.9 ms	149.6 / 149.5 ms
string_dictionary	74.4 / 74.4 ms	73.8 / 73.6 ms
string_ree_95pct_null	31.3 / 31.2 ms	31.1 / 30.9 ms
string_non_null	176.4 / 179.7 ms	175.7 / 176.0 ms
large_string_non_null	58.1 / 58.0 ms	58.1 / 57.9 ms
list_string_ree	137.7 / 137.6 ms	138.9 / 138.9 ms

Every case except the sparse one is within noise. perf on byte_array_sparse_99pct_null: byte_budget_sub_batch (with the inlined def-level scan) drops from 10.2% to 0.6% of samples, and count_values_within_byte_budget_gather rises from 0.9% to 1.6%.

Are there any user-facing changes?

No.

🤖 Generated with Claude Code

adriangb · 2026-10-30T01:33:11Z

run benchmark arrow_writer

adriangbot · 2026-10-30T01:33:20Z

🤖 Benchmark starting (GKE) | trigger

Target: arrow_writer

Sharding factor: 1 (1 workers).

Comparing perf-byte-budget-precheck (3928536) to 9ffd63d (merge-base) diff

Run configuration

run benchmark arrow_writer
shards: 1

Results will be posted when all workers finish.

File an issue against this benchmark runner

adriangb · 2026-10-30T02:34:55Z

run benchmark arrow_writer

adriangbot · 2026-10-30T02:34:59Z

🤖 Benchmark starting (GKE) | trigger

Target: arrow_writer

Sharding factor: 1 (1 workers).

Comparing perf-byte-budget-precheck (f17e631) to 9ffd63d (merge-base) diff

Run configuration

run benchmark arrow_writer
shards: 1

Results will be posted when all workers finish.

File an issue against this benchmark runner

adriangb · 2026-10-30T02:56:42Z

run benchmark arrow_writer
shards: 4

adriangbot · 2026-10-30T03:33:54Z

🤖 Benchmark failed or incomplete (GKE) | trigger

Target: arrow_writer

Comparing perf-byte-budget-precheck (3928536) to 9ffd63d (merge-base) diff

Run configuration

run benchmark arrow_writer
shards: 1

Errors / missing shards

arrow_writer — shard 1/1: Benchmark for [this request](https://github.com/apache/arrow-rs/pull/11997#issuecomment-6512736651) hit the 7200s job deadline before finishing.

Benchmarks requested: `arrow_writer`

<details><summary>Kubernetes message</summary>

```
Job was active longer than specified deadline
```

</details>

---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner

Per-runner information

arrow_writer — shard 1/1

Node: gk3-benchmark-cluster-nap-1fzug0kq-50073869-jzxw

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6512736651-4131-k6w5t 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

No completed measurement resource samples.

File an issue against this benchmark runner

adriangbot · 2026-10-30T04:01:13Z

🤖 Benchmark starting (GKE) | trigger

Target: arrow_writer

Sharding factor: 4 (4 workers).

Comparing perf-byte-budget-precheck (318bf4c) to 9ffd63d (merge-base) diff

Run configuration

run benchmark arrow_writer
shards: 4

Results will be posted when all workers finish.

File an issue against this benchmark runner

adriangbot · 2026-10-30T04:35:32Z

🤖 Benchmark failed or incomplete (GKE) | trigger

Target: arrow_writer

Comparing perf-byte-budget-precheck (f17e631) to 9ffd63d (merge-base) diff

Run configuration

run benchmark arrow_writer
shards: 1

Errors / missing shards

arrow_writer — shard 1/1: Benchmark for [this request](https://github.com/apache/arrow-rs/pull/11997#issuecomment-6513550211) hit the 7200s job deadline before finishing.

Benchmarks requested: `arrow_writer`

<details><summary>Kubernetes message</summary>

```
Job was active longer than specified deadline
```

</details>

---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner

Per-runner information

arrow_writer — shard 1/1

Node: gk3-benchmark-cluster-nap-6uf8krqf-51bf6dd9-z9n6

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6513550211-4132-pq5j2 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

No completed measurement resource samples.

File an issue against this benchmark runner

adriangbot · 2026-10-30T05:42:33Z

🤖 Benchmark completed (GKE) | trigger

Target: arrow_writer

Comparing perf-byte-budget-precheck (318bf4c) to 9ffd63d (merge-base) diff

Run configuration

run benchmark arrow_writer
shards: 4

Comparison results: View the complete native critcmp output.

The comparison is published in an unlisted Gist because the original inline report exceeded GitHub's comment limit.

Per-runner information

arrow_writer — shard 1/4

Node: gk3-benchmark-cluster-nap-jeel64w1-66f44736-l628

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6513830333-4137-ccmpb 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	1745.4s
Peak memory	2.1 GiB
Avg memory	196.1 MiB
CPU user	1708.6s
CPU sys	31.5s
Peak spill	0 B

branch

Metric	Value
Wall time	1745.4s
Peak memory	1.9 GiB
Avg memory	193.9 MiB
CPU user	1705.9s
CPU sys	33.2s
Peak spill	0 B

arrow_writer — shard 2/4

Node: gk3-benchmark-cluster-nap-1fzug0kq-50073869-h5nx

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6513830333-4138-pksqp 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	1780.4s
Peak memory	2.2 GiB
Avg memory	352.9 MiB
CPU user	1724.6s
CPU sys	53.5s
Peak spill	0 B

branch

Metric	Value
Wall time	1805.4s
Peak memory	2.2 GiB
Avg memory	346.0 MiB
CPU user	1748.4s
CPU sys	50.7s
Peak spill	0 B

arrow_writer — shard 3/4

Node: gk3-benchmark-cluster-nap-6uf8krqf-51bf6dd9-z9n6

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6513830333-4139-cnjns 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	1925.4s
Peak memory	2.1 GiB
Avg memory	288.7 MiB
CPU user	1877.7s
CPU sys	41.7s
Peak spill	0 B

branch

Metric	Value
Wall time	1905.4s
Peak memory	2.1 GiB
Avg memory	286.1 MiB
CPU user	1861.4s
CPU sys	37.9s
Peak spill	0 B

arrow_writer — shard 4/4

Node: gk3-benchmark-cluster-nap-1fzug0kq-c69560ca-b6ps

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6513830333-4140-b5bn4 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	1765.4s
Peak memory	2.0 GiB
Avg memory	224.7 MiB
CPU user	1729.3s
CPU sys	30.7s
Peak spill	0 B

branch

Metric	Value
Wall time	1740.4s
Peak memory	2.0 GiB
Avg memory	224.4 MiB
CPU user	1707.2s
CPU sys	28.8s
Peak spill	0 B

File an issue against this benchmark runner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Raw ColumnWriter callers may pad a contiguous values slice with placeholder ByteArrays past the values their levels reference; sizing them panics. Restrict the lookahead window to gather indices, which always name real values, and keep counting the chunk's values first on the contiguous path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

adriangb · 2026-11-03T04:08:13Z

run benchmark arrow_writer

shards: 4

adriangbot · 2026-11-03T04:08:18Z

🤖 Benchmark starting (GKE) | trigger

Target: arrow_writer

Sharding factor: 4 (4 workers).

Comparing perf-byte-budget-precheck (bf51fa4) to b231cf1 (merge-base) diff

Run configuration

run benchmark arrow_writer
shards: 4

Results will be posted when all workers finish.

File an issue against this benchmark runner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

adriangbot · 2026-11-03T05:16:10Z

🤖 Benchmark completed (GKE) | trigger

Target: arrow_writer

Comparing perf-byte-budget-precheck (bf51fa4) to b231cf1 (merge-base) diff

Run configuration

run benchmark arrow_writer
shards: 4

Comparison results: View the complete native critcmp output.

The comparison is published in an unlisted Gist because the original inline report exceeded GitHub's comment limit.

Per-runner information

arrow_writer — shard 1/4

Node: gk3-benchmark-cluster-nap-17yz8u0s-c78dad10-hp78

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6595202522-4291-f4plb 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	1740.4s
Peak memory	2.1 GiB
Avg memory	196.0 MiB
CPU user	1706.6s
CPU sys	32.4s
Peak spill	0 B

branch

Metric	Value
Wall time	1735.4s
Peak memory	2.1 GiB
Avg memory	195.2 MiB
CPU user	1700.2s
CPU sys	31.9s
Peak spill	0 B

arrow_writer — shard 2/4

Node: gk3-benchmark-cluster-nap-r5b1bsxo-0e1bca2a-sz8x

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6595202522-4292-2m9bl 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	1790.4s
Peak memory	2.1 GiB
Avg memory	353.5 MiB
CPU user	1737.3s
CPU sys	49.8s
Peak spill	0 B

branch

Metric	Value
Wall time	1790.4s
Peak memory	2.2 GiB
Avg memory	350.3 MiB
CPU user	1736.5s
CPU sys	47.7s
Peak spill	0 B

arrow_writer — shard 3/4

Node: gk3-benchmark-cluster-nap-17yz8u0s-dc2beb66-4t8b

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6595202522-4293-mpwnj 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	1905.4s
Peak memory	2.2 GiB
Avg memory	286.7 MiB
CPU user	1861.7s
CPU sys	37.6s
Peak spill	0 B

branch

Metric	Value
Wall time	1905.4s
Peak memory	2.3 GiB
Avg memory	287.2 MiB
CPU user	1861.9s
CPU sys	38.4s
Peak spill	0 B

arrow_writer — shard 4/4

Node: gk3-benchmark-cluster-nap-r5b1bsxo-7fe87dcb-5l4n

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6595202522-4294-5nwgb 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	1750.4s
Peak memory	1.9 GiB
Avg memory	224.1 MiB
CPU user	1720.1s
CPU sys	27.0s
Peak spill	0 B

branch

Metric	Value
Wall time	1735.4s
Peak memory	2.1 GiB
Avg memory	225.8 MiB
CPU user	1706.0s
CPU sys	27.1s
Peak spill	0 B

File an issue against this benchmark runner

github-actions Bot added the parquet Changes to the parquet crate label Oct 30, 2026

adriangb force-pushed the perf-byte-budget-precheck branch from 3928536 to f17e631 Compare October 30, 2026 02:34

adriangb changed the title ~~perf(parquet): skip def-level scan when a chunk's bytes plainly fit~~ perf(parquet): count fitting values before scanning def levels Oct 30, 2026

adriangb mentioned this pull request Oct 30, 2026

fix(parquet): cap byte-budget level windows by value count #11996

Merged

adriangb and others added 3 commits November 2, 2026 20:54

perf(parquet): count fitting values before scanning def levels

0da4911

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(parquet): cover byte-budget chunker window decisions

5f52b9b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

adriangb force-pushed the perf-byte-budget-precheck branch from 193e106 to bf51fa4 Compare November 3, 2026 02:55

adriangb marked this pull request as ready for review November 3, 2026 04:06

adriangbot mentioned this pull request Nov 3, 2026

Reuse dictionary hashes in repartition and aggregation apache/datafusion#27032

Draft

adriangb and others added 2 commits November 2, 2026 22:30

docs(parquet): state the count_values_within_byte_budget prefix contract

1f741a6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(parquet): split long byte-budget chunker comments

cf99e7a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Navigation Menu

Repository navigation

perf(parquet): count fitting values before scanning def levels - #11997

perf(parquet): count fitting values before scanning def levels#11997
adriangb wants to merge 5 commits into
apache:mainfrom
pydantic:perf-byte-budget-precheck

adriangb commented Oct 30, 2026 •

edited

Loading

Uh oh!

adriangb commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangb commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangb commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangb commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

adriangb commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

adriangb commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangb commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangb commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangbot commented Oct 30, 2026

Uh oh!

adriangb commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adriangb commented Oct 30, 2026 •

edited

Loading