[C++][Compute] Casting an all-null list array read from IPC to fixed_size_list segfaults (Take with out-of-bounds index on an empty child) #54570

Describe the bug, including details regarding any error messages, version, and platform.

Describe the bug, including details regarding any error messages, version, and platform.

Casting a list/large_list/list_view array to fixed_size_list crashes the process with a segmentation fault when the array's null slots do not have the target length and the child (values) array is empty with no data buffer. A null list read back from IPC is enough to trigger it:

Reproducer

import pyarrow as pa
import pyarrow.ipc as ipc

t = pa.table({"c": pa.array([None], pa.list_(pa.int64()))})
sink = pa.BufferOutputStream()
with ipc.new_stream(sink, t.schema) as w:
    w.write_table(t)
col = ipc.open_stream(sink.getvalue()).read_all()["c"]
col.validate(full=True)                              # passes

print(col.cast(pa.list_(pa.int64(), 3)))            # expected [null]

Output on main:

Segmentation fault (core dumped)

The IPC file format crashes too, and so do arrays built with from_buffers whose zero-length child has a null data buffer (spec-valid for a zero-length array):

child = pa.Array.from_buffers(pa.int64(), 0, [None, None])
arr = pa.ListArray.from_buffers(pa.list_(pa.int64()), 1,
                                [pa.py_buffer(b"\x00"), pa.array([0, 0], pa.int32()).buffers()[1]],
                                children=[child])
arr.cast(pa.list_(pa.int64(), 3))                   # segfault

We first hit this on a Polars-produced large_list<timestamp[us]> (Polars to_arrow()): a chunk holding a single null list with an empty child.

Backtrace (pyarrow 25.0.1, gdb):

#0  arrow::compute::internal::(anonymous)::FixedWidthTakeImpl<unsigned long, std::integral_constant<int, 64>, ...>::Exec(...)
#1  arrow::compute::internal::FixedWidthTakeExec(...)
...
#11 arrow::compute::Take(arrow::Datum const&, arrow::Datum const&, arrow::compute::TakeOptions const&, arrow::compute::ExecContext*)
#12 arrow::compute::internal::(anonymous)::CastVarToFixedList<arrow::LargeListType>::Exec(...)

Versions: pyarrow 27.0.0.dev13 (nightly built from main at 9f03413), Python 3.12, Linux x86_64 (Ubuntu 22.04, glibc 2.35). The released wheels 15.0.2, 17.0.0, 19.0.1, 21.0.0, 23.0.0, 25.0.1 and 26.0.0 crash the same way.

Where it comes from

• When a null slot does not have length list_size, CastVarToFixedList::Exec rebuilds the values with Take. BuildTakeIndices emits the valid index 0 for every element of a null slot ("just fill in the null slots with first value", scalar_cast_nested.cc#L230-L251).
• Take is then called with TakeOptions::NoBoundsCheck() (scalar_cast_nested.cc#L279-L286). If the child is empty, index 0 is out of bounds. With a null data buffer, FixedWidthTakeImpl dereferences a null pointer. With an allocated buffer, it reads padding, and the result is hidden because the slot is null.
• pc.take on the same empty child with null indices ([None, None, None]) works, so the bad indices come from the cast kernel.
• The kernel is registered for list, large_list, list_view, large_list_view and map sources (scalar_cast_nested.cc#L516-L521).

Possible fixes:

• Emit a null index for the elements of a null slot (append nulls in BuildTakeIndices) instead of 0, so Take never reads the child for them.
• Or, when the cast child has length 0, build the output values with MakeArrayOfNulls(child_type, length * list_size) instead of calling Take.

Impact: any process that casts list columns to fixed_size_list (for example to build embedding or tensor columns) crashes on input read from IPC or Arrow Flight, or received over the C Data Interface from another library, as soon as one batch holds only null lists. The crash cannot be caught from Python.

Component(s)

C++, Python

Component(s)

C++