-
Notifications
You must be signed in to change notification settings - Fork 12.9k
Tempo: Add support for TraceQL Metrics exemplars #96859
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
/deploy-to-hg |
|
Error building instance: Contact #proj-ephemeral-hg-instances if it is not a compile error. Logs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is really cool, nice job 👍
Tooltips seems to get stuck in the UI after hovering over a few exemplars.
If I click around a few times on the various tabs + RED metric cards, eventually the diamonds turn into x's.
Upon changing tabs, all of the exemplars disappear from the rate metric card.
Do you know for certain why some exemplars are darker than others?
The upper/lower case combinations between key/value pairs in the tooltip are a little confusing. I can see the reasoning but this will not be as clear to users. We can revisit this with some UX help in the New Year.
How is the value from the exemplar reflected in the trace? Is it just the case that this trace was recorded during the course of this rate (error) of requests? (this is to solidify my own understanding)
public/app/plugins/datasource/tempo/traceql/TempoQueryBuilderOptions.tsx
Show resolved
Hide resolved
We don't have much control over the rendering portion here since the exemplars are being rendered by a plugin of the Timeseries and Heatmap visualizations. So I can't really explain why some of these things happen, but if they do we can either try to fix them in another PR or create issues for the owner of the vizualisations to take a look. However, these exemplars are the same as the Prometheus exemplars.
I'll take a look!
Yes, Tempo will take a sample of the traces that generated these metrics and link them to the series. Tempo sends the trace ID and a few bits of metadata (like the timestamp and value) so the exemplar can be rendered in the right point of the graph. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, looks great! 👍
As discussed, there are still a few issues in the app which Andre plans to work on soon.
What is this feature?
This PR adds support for exemplars in TraceQL metrics queries for the Tempo data source. The exemplars were already in the response but Grafana was ignoring them and returning only the series. This PR transforms the exemplars from the Tempo format to DataFrames which can be used by Grafana panels, such as the TimeSeries or Histogram panels.
See the documentation on how to enable exemplars in Tempo.
Why do we need this feature?
Exemplars provide a simple way to view discrete traces while looking at aggregate data through TraceQL metrics queries.
Who is this feature for?
Users of the Tempo datasource.
Which issue(s) does this PR fix?:
Fixes #96290
Special notes for your reviewer:
Please check that: