Repository navigation
[C++] drop_null kernel ignores nulls when values is an union or a REE array #43851
Description
Describe the bug, including details regarding any error messages, version, and platform.
Unions and run-end-encoded don't have validity bitmaps (buffers[0]), so they are sometimes treated as "array without nulls". This happens in the drop_null function: so even when unions have nulls, the input -- values -- is returned unfiltered.
Component(s)
C++
Activity
xgbtxy commentedon Oct 20, 2026
I reproduced the union-array part on Windows x64 with PyArrow 26.0.0:
drop_nullretains logical nulls in both dense and sparse unions, including ChunkedArray, RecordBatch and Table inputs. Explicitis_valid+ filtering produces the expected rows.I am preparing a focused fix for unions across those four containers, with sliced inputs and mixed ordinary/union validity in the regressions. Run-end encoded selection is a separate limitation:
array_filterandarray_takecurrently lack REE-value kernels, so this patch will not claim to close the whole issue. Existing dictionary null semantics will remain unchanged.This work is AI-assisted; I will include the executed tests and limitations with the patch.