Skip to content

Alerting: Add state history backend to write ALERTS metric #104361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 18, 2025
Merged

Alerting: Add state history backend to write ALERTS metric #104361

merged 1 commit into from
Jun 18, 2025

Conversation

alexander-akhmetov
Copy link
Contributor

@alexander-akhmetov alexander-akhmetov commented Apr 22, 2025 •
edited
Loading

What is this feature?

This PR implements a new Prometheus historian backend that allows Grafana alerting to write alert state history as Prometheus-compatible ALERTS metrics to remote Prometheus-compatible data sources.

The metric includes a few additional labels:

  • grafana_alertstate: Grafana's full alert state, more granular than Prometheus.
  • grafana_rule_uid: Grafana's alert rule UID.

Grafana states are included in the grafana_alertstate label also mapped to Prometheus-compatible alertstate values:

Grafana alert state alertstate grafana_alertstate
Alerting firing alerting
Recovering firing recovering
Pending pending pending
Error firing error
NoData firing nodata
Normal (no metric emitted) (no metric emitted)

Configuration

The backend can be enabled in the configuration file. For example:

[unified_alerting.state_history]
enabled = true

backend = prometheus
prometheus_metric_name = ALERTS
prometheus_target_datasource_uid = my-mimir
prometheus_write_timeout = 10s

Because the prometheus backend does not support querying, it will fail when it is configured as a primary backend.

Or together with Loki state history:

[unified_alerting.state_history]
enabled = true
backend = multiple

primary = loki
loki_remote_url = http://localhost:3100

secondaries = prometheus
prometheus_metric_name = ALERTS
prometheus_target_datasource_uid = 123
prometheus_write_timeout = 10s

Example:

Please check that:

  • It works as expected from a user's perspective.
  • If this is a pre-GA feature, it is behind a feature toggle.
  • The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

@github-project-automation github-project-automation bot moved this to In review in Alerting Apr 22, 2025
@alexander-akhmetov alexander-akhmetov force-pushed the alexander-akhmetov/alerts-writer branch from 0030f5d to 01a9829 Compare May 21, 2025 15:31
@alexander-akhmetov alexander-akhmetov force-pushed the alexander-akhmetov/alerts-writer branch 2 times, most recently from b46d8fb to 4a41ed7 Compare May 30, 2025 23:18
@alexander-akhmetov alexander-akhmetov force-pushed the alexander-akhmetov/alerts-writer branch 9 times, most recently from fec8435 to 28ddfe0 Compare June 3, 2025 19:58
@alexander-akhmetov alexander-akhmetov changed the title Alerting: Add metrics writer to generate ALERTS metric Alerting: Add state history backend to write ALERTS metric Jun 5, 2025
@alexander-akhmetov alexander-akhmetov force-pushed the alexander-akhmetov/alerts-writer branch 4 times, most recently from 0f3aa40 to 706f805 Compare June 6, 2025 11:21
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 6, 2025
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 6, 2025
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 6, 2025
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 6, 2025
@grafana grafana deleted a comment from github-actions bot Jun 6, 2025
@grafana grafana deleted a comment from github-actions bot Jun 6, 2025
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 6, 2025
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 10, 2025
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 10, 2025
@alexander-akhmetov alexander-akhmetov force-pushed the alexander-akhmetov/alerts-writer branch from 1273b27 to c065846 Compare June 10, 2025 13:02
@alexander-akhmetov alexander-akhmetov marked this pull request as ready for review June 10, 2025 14:40
@alexander-akhmetov alexander-akhmetov requested review from a team as code owners June 10, 2025 14:40
@github-actions github-actions bot added this to the 12.1.x milestone Jun 10, 2025
@alexander-akhmetov alexander-akhmetov force-pushed the alexander-akhmetov/alerts-writer branch 3 times, most recently from 656c843 to fc35318 Compare June 10, 2025 15:15
@alexander-akhmetov alexander-akhmetov force-pushed the alexander-akhmetov/alerts-writer branch from fc35318 to c4fa287 Compare June 10, 2025 16:51
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 10, 2025
@grafana grafana deleted a comment from ephemeral-instances-bot bot Jun 10, 2025
@alexander-akhmetov
Copy link
Contributor Author

/deploy-to-hg

@ephemeral-instances-bot
Copy link

  • Preparing your instance. A comment containing your instance's url will be added to this PR when the instance is ready.
  • Your instance will be ready in ~10 minutes. Follow the workflow progress
  • Slack channel: #proj-ephemeral-hg-instances
  • Building instance with alexander-akhmetov/alerts-writer oss branch and main enterprise branch. How to choose a branch

@ephemeral-instances-bot
Copy link

@stevesg
Copy link
Contributor

stevesg commented Jun 10, 2025 •
edited
Loading

This is a very neat approach, I like it, will review properly shortly

stevesg
stevesg approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@stevesg stevesg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alexander-akhmetov alexander-akhmetov merged commit ad683f8 into main Jun 18, 2025
83 checks passed
@alexander-akhmetov alexander-akhmetov deleted the alexander-akhmetov/alerts-writer branch June 18, 2025 05:17
@github-project-automation github-project-automation bot moved this from In review to Done in Alerting Jun 18, 2025
@alexander-akhmetov alexander-akhmetov mentioned this pull request Jun 18, 2025
Closed
3 tasks
@ppcano ppcano mentioned this pull request Jun 19, 2025
Merged
1 task
This was referenced Jun 20, 2025
Merged
Merged
m88deng pushed a commit to m88deng/grafana that referenced this pull request Jun 30, 2025
@alexander-akhmetov @m88deng
Alerting: Add state history backend to write ALERTS metric (grafana#1…
5994e12 
…04361)

**What is this feature?**

This PR implements a new Prometheus historian backend that allows Grafana alerting to write alert state history as Prometheus-compatible `ALERTS` metrics to remote Prometheus-compatible data sources.

The metric includes a few additional labels:

* `grafana_alertstate`: Grafana's full alert state, more granular than Prometheus.
* `grafana_rule_uid`: Grafana's alert rule UID.

Grafana states are included in the `grafana_alertstate` label also mapped to Prometheus-compatible `alertstate` values:

| Grafana alert state | `alertstate`          | `grafana_alertstate`  |
|---------------------|-----------------------|-----------------------|
| `Alerting`          | `firing`              | `alerting`            |
| `Recovering`        | `firing`              | `recovering`          |
| `Pending`           | `pending`             | `pending`             |
| `Error`             | `firing`              | `error`               |
| `NoData`            | `firing`              | `nodata`              |
| `Normal`            | _(no metric emitted)_ | _(no metric emitted)_ |
YannickAlex07 pushed a commit to YannickAlex07/grafana that referenced this pull request Jul 11, 2025
@alexander-akhmetov @YannickAlex07
Alerting: Add state history backend to write ALERTS metric (grafana#1…
bb106ed 
…04361)

**What is this feature?**

This PR implements a new Prometheus historian backend that allows Grafana alerting to write alert state history as Prometheus-compatible `ALERTS` metrics to remote Prometheus-compatible data sources.

The metric includes a few additional labels:

* `grafana_alertstate`: Grafana's full alert state, more granular than Prometheus.
* `grafana_rule_uid`: Grafana's alert rule UID.

Grafana states are included in the `grafana_alertstate` label also mapped to Prometheus-compatible `alertstate` values:

| Grafana alert state | `alertstate`          | `grafana_alertstate`  |
|---------------------|-----------------------|-----------------------|
| `Alerting`          | `firing`              | `alerting`            |
| `Recovering`        | `firing`              | `recovering`          |
| `Pending`           | `pending`             | `pending`             |
| `Error`             | `firing`              | `error`               |
| `NoData`            | `firing`              | `nodata`              |
| `Normal`            | _(no metric emitted)_ | _(no metric emitted)_ |
@BrewTestBot BrewTestBot mentioned this pull request Jul 23, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stevesg stevesg stevesg approved these changes

@torkelo torkelo Awaiting requested review from torkelo torkelo is a code owner

Assignees
No one assigned
Labels
add to changelog area/alerting Grafana Alerting area/backend no-backport Skip backport of PR
Projects
Alerting
Status: Done
Milestone
12.1.x
Development

Successfully merging this pull request may close these issues.
2 participants
@alexander-akhmetov @stevesg