Overflow strings allocations: avoid rounding up memory allocated per overflow string - when reading "small" overflow strings place them directly into the vector instead #16283
+21
−10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…overflow string - when reading "small" overflow strings place them directly into the vector instead
2 tasks
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
Mar 7, 2025
Overflow strings allocations: avoid rounding up memory allocated per overflow string - when reading "small" overflow strings place them directly into the vector instead (duckdb/duckdb#16283)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #16277
Overflow strings are not necessarily extremely large - they can be relatively small (e.g. 8~16KB each) - since also for those small-ish strings we cannot fit 2048 onto a single page. In the current code for reading overflow strings from disk we allocate at least a block per overflow string - but for these small overflow strings that is very wasteful. This PR modifies this behavior to place overflow strings that are smaller than a block directly into the string vector instead of allocating a dedicated memory block for each of them.