[mikamikasuki]

Rationale for this change

Fixes #50482. A Python-backed dataset scan can return ArrowInvalid while CPU-pool workers still retain Python-backed resources. During interpreter finalization, the main thread waits for pool shutdown as a worker attempts to attach to Python, so the process can hang or abort.

What changes are included in this PR?

Register an atexit callback that releases the GIL while waiting for the CPU thread pool to become idle before interpreter finalization. Add a subprocess regression test for an FSSpec-backed dataset containing one malformed Parquet fragment.

Are these changes tested?

• Built Arrow and editable PyArrow from this branch on macOS.
• pyarrow/tests/test_misc.py: 123 passed, 1 skipped on CPython 3.12.15 and 3.14.8.
• The targeted exit regression passed 12/12 runs on each Python version.
• On PyArrow 26.0.0, the baseline timed out in 8/8 runs on CPython 3.12.15 and 7/8 runs on 3.14.8.

Are there any user-facing changes?

The malformed dataset continues to raise ArrowInvalid; after the error, interpreter shutdown now waits for Python-backed cleanup to finish.

This PR contains a "Critical Fix." It prevents the reported process-exit hang or abort when worker cleanup attaches to Python during interpreter finalization.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

AI assistance was used to trace the thread-pool shutdown path, implement the wait callback and regression test, and draft this description. The generated changes were reviewed and built on CPython 3.12.15 and 3.14.8.

• GitHub Issue: [Python] Interpreter hangs at exit after pq.read_table(python_file_object) - thread pool teardown deadlock #50482

[github-actions]

⚠️ GitHub issue #50482 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #50482 has no components, please add labels for components.

[jackboyla]

I found a pending-I/O case while investigating huggingface/datasets#8793: after a Parquet prebuffer read error, the CPU pool can be idle while other reads still hold Python callbacks. Both threaded and nonthreaded decoding reproduce the fatal shutdown error on this PR's current head.

I opened an extension against your branch: mikamikasuki#1. It waits for the default CPU and I/O pools to be idle together, since tasks can submit work between pools. The new subprocess regressions fail on the CPU-only implementation and pass with the extension in full source builds.

A reconstructed two-source Hub workload also reproduced the fatal error after 1.509 GB on the CPU-only build with unmodified Datasets. With both the Datasets iterator-ownership fix and this extension, two matching full-size runs exited cleanly; all output hashes matched.

Would you be open to incorporating the extension into this PR? The test and implementation changes are one commit on top of your branch.