Remove sampling functionality #16010
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
af44ef7 to
5a32dc5
Compare
|
Does this mean that y'all are removing sampling altogether? |
No, this is unrelated to the sample clause - this is about gathering samples as part of the table statistics to guide optimization. While the gathering part was implemented in this version the optimizer is not using it yet so this should not have any user-facing effect. |
Mytherin
added a commit
that referenced
this pull request
Feb 4, 2025
Following up on #16010 I was wondering if too much memory was used during regular sampling as well. Turns out there is and it is because we over sample. If we have a reservoir sample of 5 rows (either by sampling 0.00005% or 5 rows) then the capacity of the reservoir is calculated to be the following ``` = sample-count + (FIXED_SAMPLE_SIZE_MULTIPLIER * FIXED_SAMPLE_SIZE) = 5 + 10 * 2048 ``` The 10*2048 is quite large compared to the 5 rows. For percentage samples the extra allocations can add up since one sample is created per 100000 rows. The updated formula is ``` = sample-count + (FIXED_SAMPLE_SIZE_MULTIPLIER * MinValue<idx_t>(FIXED_SAMPLE_SIZE, sample_count)) = 5 + 10 * 5 ``` which reduces the extra space by 400x, which will also reduce the memory pressure during sampling.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses https://github.com/duckdblabs/duckdb-internal/issues/4110
Sampling has bad memory management right now, which is causing larger ingestions to fail. In the interest of releasing early next week, it's probably best that it is removed and later added when memory issues are resolved