Skip to content

[Parquet][Python] pq.read_table should allow passing MemoryPool #54637

Open

Description

@pitrou

Describe the enhancement requested

pq.read_table currently doesn't allow passing a memory pool argument, even though reading a Parquet file can be very allocation-heavy.

Component(s)

Python, Parquet

Activity

  1. changed the title [-][Python] pq.read_table should allow passing MemoryPool[/-] [+][Parquet][Python] pq.read_table should allow passing MemoryPool[/+] on Oct 19, 2026
  2. omkar-334 commented on Oct 19, 2026

    @omkar-334

    @pitrou can i work on this? Id like to give this a shot

  3. akshat-lakhera commented on Oct 19, 2026

    @akshat-lakhera

    take

    I'd like to work on this. My plan is to add a memory_pool argument to pq.read_table (and ParquetDataset / ParquetFile.read where it applies) and pass it down to the Parquet reader, with a test using a proxy memory pool to check that the allocations go through it. I'll open a PR once I have it working.

  4. pitrou commented on Oct 19, 2026

    @pitrou
    MemberAuthor

    @omkar-334 Feel free to work on this, of course.

    @akshat-lakhera You came slightly later, sorry. Also, please avoid posting AI-generated messages.

  5. akshat-lakhera commented on Oct 19, 2026

    @akshat-lakhera

    Okay @pitrou, will browse some other issues. And will not AI generated messages from now on ^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions