Implement DELTA_BINARY_PACKED compression in Parquet writer
#14257
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Regression test failures are unrelated |
|
Thanks! Looks great |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As the title explains, this PR implements
DELTA_BINARY_PACKEDcompression for the Parquet writer. I've reused our own bitpacking code to achieve this. I've reworked the reader to also use our own bitpacking code. The compression ratio for integral columns should be much better now.I've added this on a branch where I was enabling ARM Neon SIMD instructions for Snappy on AArch64. I found out that Neon is always available/enabled on AArch64! This means we can enable it in Snappy to enjoy faster (de-)compression on this CPU architecture. For x86 it's much harder to enable SIMD, as it is non-portable to older CPUs. ARM is relatively young in that sense, and we can safely enable it.