Skip to content

Conversation

adrapereira
Copy link
Contributor

@adrapereira adrapereira commented Nov 21, 2024

What is this feature?

This PR adds support for exemplars in TraceQL metrics queries for the Tempo data source. The exemplars were already in the response but Grafana was ignoring them and returning only the series. This PR transforms the exemplars from the Tempo format to DataFrames which can be used by Grafana panels, such as the TimeSeries or Histogram panels.

See the documentation on how to enable exemplars in Tempo.

Why do we need this feature?

Exemplars provide a simple way to view discrete traces while looking at aggregate data through TraceQL metrics queries.

Who is this feature for?

Users of the Tempo datasource.

Which issue(s) does this PR fix?:

Fixes #96290

Special notes for your reviewer:

Please check that:

  • It works as expected from a user's perspective.
  • If this is a pre-GA feature, it is behind a feature toggle.
  • The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

@adrapereira
Copy link
Contributor Author

/deploy-to-hg

@ephemeral-instances-bot
Copy link

  • Preparing your instance. A comment containing your instance's url will be added to this PR when the instance is ready.
  • Your instance will be ready in ~10 minutes. Follow the workflow progress
  • Slack channel: #proj-ephemeral-hg-instances
  • Building instance with andre/traceql-metrics-exemplars oss branch and main enterprise branch. How to choose a branch

@ephemeral-instances-bot
Copy link

Error building instance: Contact #proj-ephemeral-hg-instances if it is not a compile error. Logs

@mdisibio mdisibio mentioned this pull request Nov 21, 2024
3 tasks
@adrapereira adrapereira marked this pull request as ready for review December 12, 2024 14:16
@adrapereira adrapereira requested a review from a team as a code owner December 12, 2024 14:16
@github-actions github-actions bot added this to the 11.5.x milestone Dec 12, 2024
Copy link
Contributor

@joey-grafana joey-grafana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really cool, nice job 👍

Tooltips seems to get stuck in the UI after hovering over a few exemplars.
Screenshot 2024-12-20 at 13 14 54

If I click around a few times on the various tabs + RED metric cards, eventually the diamonds turn into x's.
Screenshot 2024-12-20 at 13 29 15

Upon changing tabs, all of the exemplars disappear from the rate metric card.
Screenshot 2024-12-20 at 13 31 00

Do you know for certain why some exemplars are darker than others?
Screenshot 2024-12-20 at 13 14 34

The upper/lower case combinations between key/value pairs in the tooltip are a little confusing. I can see the reasoning but this will not be as clear to users. We can revisit this with some UX help in the New Year.
Screenshot 2024-12-20 at 13 23 41

How is the value from the exemplar reflected in the trace? Is it just the case that this trace was recorded during the course of this rate (error) of requests? (this is to solidify my own understanding)
Screenshot 2024-12-20 at 13 16 07

@adrapereira
Copy link
Contributor Author

This is really cool, nice job 👍

Tooltips seems to get stuck in the UI after hovering over a few exemplars.

If I click around a few times on the various tabs + RED metric cards, eventually the diamonds turn into x's.

Do you know for certain why some exemplars are darker than others?

The upper/lower case combinations between key/value pairs in the tooltip are a little confusing. I can see the reasoning but this will not be as clear to users. We can revisit this with some UX help in the New Year.

We don't have much control over the rendering portion here since the exemplars are being rendered by a plugin of the Timeseries and Heatmap visualizations. So I can't really explain why some of these things happen, but if they do we can either try to fix them in another PR or create issues for the owner of the vizualisations to take a look. However, these exemplars are the same as the Prometheus exemplars.

Upon changing tabs, all of the exemplars disappear from the rate metric card.

I'll take a look!

How is the value from the exemplar reflected in the trace? Is it just the case that this trace was recorded during the course of this rate (error) of requests? (this is to solidify my own understanding)

Yes, Tempo will take a sample of the traces that generated these metrics and link them to the series. Tempo sends the trace ID and a few bits of metadata (like the timestamp and value) so the exemplar can be rendered in the right point of the graph.

@adrapereira adrapereira enabled auto-merge (squash) January 10, 2025 13:19
Copy link
Contributor

@joey-grafana joey-grafana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, looks great! 👍

As discussed, there are still a few issues in the app which Andre plans to work on soon.

@adrapereira adrapereira merged commit b742896 into main Jan 10, 2025
21 checks passed
@adrapereira adrapereira deleted the andre/traceql-metrics-exemplars branch January 10, 2025 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tempo: Add support for exemplars in TraceQL metrics queries
3 participants