[Python] Process aborts at interpreter exit: ~PyReadableFile takes the GIL on a ThreadPool worker during finalization (pd.read_parquet on a local path) #54536

Describe the bug, including details regarding any error messages, version, and platform.

A short-lived process that reads a local parquet file through pandas and then exits sometimes aborts with SIGABRT (exit 134), after all of its work and output are done. stderr shows terminate called without an active exception and Fatal Python error: Aborted, with no Python frame.

pandas opens a Python file handle for a local path and passes it to the dataset reader, which wraps it in a PyReadableFile. A reader task can drop the last reference on a ThreadPool worker thread while the interpreter is finalizing. OwnedRefNoGIL::~OwnedRefNoGIL then calls PyGILState_Ensure, CPython ends the thread with pthread_exit, the forced unwind passes a noexcept destructor, and std::terminate is called.

Reproduction

# write.py: run once
import pandas as pd
pd.DataFrame({"a": [1, 2], "b": [1.0, 2.0]}).to_parquet("p.parquet", index=False)

# read.py: run many times, a few in parallel
import pandas as pd, sys
pd.read_parquet(sys.argv[1], columns=["a", "b"])
print("ok")

python write.py
seq 3000 | xargs -P 4 -I{} sh -c 'python read.py p.parquet >/dev/null 2>&1 || echo $?' | sort | uniq -c

Expected: no output (every run exits 0). Observed: 38 134, that is 38 of 3000 runs abort.

The same loop with pyarrow.parquet.read_table(sys.argv[1], columns=["a", "b"]).to_pandas() in place of pd.read_parquet gives no output, so 0 of 3000 runs abort. The file is then opened natively, and no Python object is held by the reader.

Environment

• pyarrow 26.0.0 (pyarrow 25.0.1 aborts at the same rate)
• pandas 3.0.6, numpy 2.5.3
• CPython 3.12.12
• Debian 12 (bookworm), aarch64, in Docker

Stack of the aborting thread (gdb)

#0  __pthread_kill_implementation
#3  abort
#4  __gnu_cxx::__verbose_terminate_handler()
#6  std::terminate()
#7  __gxx_personality_v0
#9  _Unwind_ForcedUnwind
#10 __pthread_unwind
#12 __pthread_exit
#13 PyThread_exit_thread
#14 take_gil
#15 PyEval_RestoreThread
#16 PyGILState_Ensure
#17 arrow::py::OwnedRefNoGIL::~OwnedRefNoGIL()
#18 arrow::py::PyReadableFile::~PyReadableFile()
#19 std::_Sp_counted_base<...>::_M_release_last_use_cold()
#20 std::_Function_handler<arrow::Result<std::shared_ptr<arrow::io::RandomAccessFile>>(), arrow::dataset::FileSource::FileSource(...)>
#21 arrow::dataset::FileSource::~FileSource()
#22 arrow::dataset::ParquetFileFragment::~ParquetFileFragment()
... on a ThreadPool worker thread (ThreadPool::LaunchWorkersUnlocked)

Possible fix. Skip the decref in OwnedRefNoGIL's destructor when the interpreter is finalizing (Py_IsFinalizing()), as other C++ extensions do. Alternatively, release the scan's fragments on the calling thread before read_table returns.

Related. ARROW-16597 has the same mechanism in Flight: PyThread_exit_thread → pthread_exit while C++ code is on the stack, at interpreter exit.

Component(s)

Python