Skip to content

Conversation

@steveloughran
Copy link
Contributor

Rationale for this change

Goal: address memory leak identified with ParquetReader.readVectored

What changes are included in this PR?

Patch 1 Moves to 3.4.1 for replication, testing and IDE...it isn't for merging.

It should show the problem.

Are these changes tested?

yes, hence the report

Are there any user-facing changes?

no

Closes #3237

@steveloughran steveloughran marked this pull request as draft June 4, 2025 15:37
* lots of logging in TrackingByteBufferAllocator
* Reader adds buffers to release

Reader tests do still fail, but differently. Why?
Delete checksum file so checksum validation is bypassed,
which avoids all slicing issues.

This makes the test failure "go away" but doesn't address
the underlying issue with ChecksumFileSystem subclasses,
especially LocalFileSystem.
This makes the tests pass but doesn't address
the issue that file:// reads will return sliced subsets of buffers; this is due to how
checksums are being verified.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Buffers allocated by ParquetFileReader.readVectored() are not being released

1 participant