Skip to content

[C++][Python] Use timestamp_parsers as a fallback for CSV time columns #54651

Open

Description

@pearu

Describe the enhancement requested

CSV columns typed as time32 or time64 are parsed only from the ISO-8601 format, and ConvertOptions.timestamp_parsers applies only to timestamp columns:

import io
import pyarrow as pa
from pyarrow import csv

convert_options = csv.ConvertOptions(
    column_types={"t": pa.time32("s")}, timestamp_parsers=["%H.%M.%S"]
)
print(csv.read_csv(io.BytesIO(b"t\n14.30.00\n"), convert_options=convert_options))
pyarrow.lib.ArrowInvalid: In CSV column #0: CSV conversion error to time32[s]: invalid value '14.30.00'

The workaround is to read the column as a timestamp and cast it to a time. I propose that time columns try timestamp_parsers after the ISO-8601 parser fails, and take the time of day of the parsed timestamp, as a cast from timestamp to time does.

An implementation exists in #50146, which is being refactored into smaller PRs.

Part of #41488.

Component(s)

C++, Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions