perf(chunkenc): intern custom values for native histograms #16565
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Copy the benchmark for native histograms with exponential buckets and adopt to native histograms with custom buckets. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
The custom values are the "le" bucket boundaries of native histograms with custom buckets. They are never modified. It is ok to not copy them when iterating a chunk, just reference them. If we will ever have a function that modifies the custom values, like 'trim' for example. That function will have to make a copy on write. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
426dcba to
6c64665
Compare
|
It is an open question whether to keep counting the size of the custom values for the engine here. There's no trivial answer. In this first step the interning works per chunk. So loading 1000 samples from a series could still results in 100x custom values. And of course loading 1000 series with 1 sample will result in 1000x custom values. |
beorn7
approved these changes
May 12, 2025
There was a problem hiding this comment.
Thanks. Looks good.
Let's the size logic as is for now. It's giving us the worst case, but it's also not such a big deal compared to the total size of the histogram. When we do "perfect interning" (no duplicate custom values at all), we can remove the variable part in the size calculation.
charleskorn
added a commit
to grafana/mimir
that referenced
this pull request
May 27, 2025
charleskorn
added a commit
to grafana/mimir
that referenced
this pull request
Jun 2, 2025
* Update mimir-prometheus * Replace references to `InstanceName` removed in prometheus/prometheus#16228 * Fix native histogram mangling behaviour to be compatible with prometheus/prometheus#16565 * Make build tags consistent with Prometheus * Update MQE tests to match upstream * Reinstate test cases that were skipped due to upstream issues * Bring in changes from grafana/mimir-prometheus#880 * Add changelog entries * Regenerate OTLP files * Fix breaking change * Update upstream tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #13490 . Although we can do more if we implement not losing custom values in the iterator between iterator reuse.