Currently, incoming webhooks will return an HTTP 4xx or 5xx error under any of these circumstances:

1. The incoming payload is invalid
2. The handler encountered an error trying to generate notifications from the payload
3. Any of the notifications failed to deliver for any reason

This behavior seems ideal for scenarios 1 and 2, but not for 3. We should only return errors if the payload was malformed or we just couldn't handle it. Other cases (such as delivery failures in scenario 3) are not the webhook's fault and should therefore not be reported as errors to them. (We should instead of our own internal mechanisms to retry delivery, log failures, etc)

The current behavior in scenario 3 is especially problematic for senders like Argo CD, which will reattempt delivery of the webhook to Mailroom multiple times if it sees an HTTP 5xx response code.

Let's therefore change this behavior to:

• Always return HTTP 202 Accepted whenever we successfully parse the payload and generate some pending notifications
• Rely on Mailroom's error logging to indicate delivery failures instead of exposing errors to our callers