-
Notifications
You must be signed in to change notification settings - Fork 272
Description
What problem would you like to solve? Please describe:
When using webhook as a user notification method, the "Name" field of the configured/alerting Service is not included in the data supplied to the webhook. It would be useful to know which service alerts are originating from without having to rely on the alert's data containing the service name.
In a particular use case, we utilize anonymous service names and a webhook which strips identifying data from the incoming alert to prevent leakage of sensitive data outside of the managed service network. We utilize a webhook to receive the alert data and scrub sensitive data out of the json before forwarding it on to an SMS service to deliver the notification to an on-call engineer. The anonymization process strips out anything like prometheus URL or service-identifying information/URLs from the json alert data, but there is nothing that seemingly links the data to the service within goalert which created the alert.
Describe the solution you'd like:
Include service name (or the UUID of the service) as a field in json sent to the notification webhook.
Describe alternatives you've considered:
Alternatives considered include bit-shifting or otherwise anonymizing the data within the alert that is already present.
Additional context:
Contextually, our webhook strips all information except for the goalert Alert ID and Alert Summary data from the incoming alert json, but correlating multiple alerts to the same service is not possible at the receiving end of the SMS messages. It would be useful for someone receiving an alert to know whether the issue is a repeat of the same "summary" from the same "service" or whether it's coming from a new "service". Also, it would be useful to know that multiple different summary alerts are coming from one service or many different services without having to login to the goalert UI. This allows for on-call engineers to properly context based on alert information without having to be logged into the goalert UI. In our use case, goalert is behind a VPN which is not available on the same mobile device receiving the SMS alerts, so the usefulness of information in the outgoing alert is necessary for accessibility to actionable information at the end user SMS device.
Imagine if someone has a service that needs a 4 hour response SLA, and one service that has a 1 hour response SLA. If both of these are alerting to the same webhook for a given on-call engineer, the engineer cannot know which SLA to meet without the context of which service sent the alert. The most useful way for us to address this would be by including the service name (in one example, these service names might be "MyService - High" or "MyService - Low").