Fix traceql exemplar distribution #5129
Conversation
|
|
|
Thank you for your contribution and for taking the time to put this well-explained proposal together! To simplify a bit, TraceQL Metrics are calculated independently in the queriers for each block, and then merged together in the query-frontend. One of the challenges with exemplar distribution over time is ensuring a fair distribution at both of these stages. While we do have bucket sampling, it doesn’t work particularly well when it comes to distribution fairness. To address the issue, we needed to change the bucketing algorithm and the approach to calculating requested exemplars from frontend to determine its fair share based on the block's time range. I’ve implemented this approach here: #5158. It avoids additional sampling steps and has linear complexity. Feel free to take a look and leave any feedback or questions on the PR. Thanks again for your interest and ideas! |
|
The issue is fixed in #5158 |
What this PR does:
Adds a new package
exemplardistthat implements an algorithm to fix the issue where TraceQL metrics exemplars cluster on one side of the visualization instead of being distributed homogeneously across the time range. This package provides a bucketing algorithm to evenly distribute exemplars across the time range while preserving their representative quality.Which issue(s) this PR fixes:
Fixes #4856
Checklist
CHANGELOG.mdupdated - Not updated since this is a standalone utility packageAdditional Notes
This PR only adds the
exemplardistpackage without modifying the core Tempo code. Direct integration into the core codebase was attempted but caused test failures due to the complex interactions with existing code.The package provides a clean API that can be used at various integration points:
This solution uses a bucketing algorithm that:
The implementation is tested to work with various distributions (uniform, left-skewed, right-skewed, clustered) and correctly improves the distribution quality in all cases.