Repository navigation
GH-53290: [Python] CSV: expose pad_short_rows parse option in Python - #54691
Open
Mr-Nobody-Anonymous wants to merge 2 commits into
Open
Mr-Nobody-Anonymous wants to merge 2 commits into
Mr-Nobody-Anonymous wants to merge 2 commits into
Conversation
…ython Add pad_short_rows to pyarrow.csv.ParseOptions, allowing short CSV rows with fewer fields than expected to be padded with null values rather than raising an ArrowInvalid parse error. * Add pad_short_rows property, init argument, equals, repr, and serialization in _csv.pyx * Expose pad_short_rows in libarrow.pxd CCSVParseOptions * Add tests in test_csv.py and test_dataset.py * Document option in docs/source/python/csv.rst
Mr-Nobody-Anonymous
requested review from
AlenkaF,
raulcd and
rok
as code owners
October 20, 2026 10:52
|
|
…_rows test coverage * Correct integer type expectation for numerical short row CSV tests * Add string data test cases for pad_short_rows * Test distinction between missing trailing fields and empty fields * Test error on excess columns when pad_short_rows=True * Test header-only input with pad_short_rows=True * Test backward-compatible unpickling of legacy 7-element ParseOptions state * Test ParseOptions equality differentiation on pad_short_rows * Add dataset reader integration test with CsvFileFormat(pad_short_rows=True)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
In #50926, pad_short_rows was implemented in the C++ CSV parser/reader (�rrow::csv::ParseOptions::pad_short_rows), allowing rows with fewer fields than expected to be padded with null values rather than failing. However, this option was not yet exposed to the Python bindings (#53290).
What changes are included in this PR?
Are these changes tested?
Yes. Added tests in est_csv.py and est_dataset.py covering parse options validation, pickling, repr, legacy unpickling, and reading short rows in serial and parallel modes.
Are there any user-facing changes?
Yes. pyarrow.csv.ParseOptions now accepts pad_short_rows (defaulting to False), allowing users to opt into padding short rows with nulls.
Was AI used for this PR?
AI assistance was used for code drafting, technical audit, and test authoring. The architecture, C++ mapping, test cases, and documentation were reviewed and verified.
PR code and description written by:
Reviewed before submission by: