Skip to content

Loki Ruler - Provide a way of accessing alert firing timestamp or current time in annotations #4980

@dwilliams782

Description

@dwilliams782

Is your feature request related to a problem? Please describe.

Note: I raised this in Slack as well as it's somewhere between a FR and a question.

We are using Loki Ruler to send alerts to Alert Manager. These alerts contain an annotation which links back to Grafana / Loki, example:

groups:
  - name: quota_exceeded
    rules:
      - alert: QUOTA_EXCEEDED
        expr: |
          sum by (project) (count_over_time({job="gcplog"} |= "QUOTA_EXCEEDED" [5m]) > 0)
        for: 1m
        annotations:
          summary: "A GCP Quota has been exceeded. See logs for details."
          loki: <loki-url>
       

The alert expression and annotations have been truncated to hide our domains but you see the point - we pass a loki URL which I would like to be able to set the from and to parameters using the time the alert fires +-15 mins so we can point to an accurate timeframe for the logs.

Describe the solution you'd like

#4603 adds some nice datetime functions like now and unixEpoch, which work well if you add them as a label. Adding them as a label to use in the URL would work but it isn't ideal as it'll interfere with the sum by aggregation, so I'd like a way to be able to call these functions.

In prometheus, you can simply query time():

{{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}

but with logQL, any attempts to use a function generates a:

parse error at line 1, col 1: syntax error: unexpected IDENTIFIER

Ideally there will be a way to generate a timestamp we can use in a similar fashion to promQL, maybe:

{{ with query "(unixEpoch now) * 1000" }}{{ . }}{{ end }} (grafana wants the from/to in ms not s)

Describe alternatives you've considered
I tried all the methods I could find for promQL but none work with logQL, and I wasn't able to iterate over the alerts to get .StartsAt.

Metadata

Metadata

Assignees

No one assigned

    Labels

    keepaliveAn issue or PR that will be kept alive and never marked as stale.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions