Prometheus metrics include dead workers #2244
Description
Bug Report
The prometheus metrics emitter never resets or deletes metrics for dead workers, so the prometheus endpoint reports container counts, volume counts, etc. for workers long after they're retired. We noticed this because one of our workers had a high container count right as it was retired, and we alert on high container counts, so we saw a phantom alert firing even though the worker it referenced didn't exist.
Not sure how to fix this with the current metrics interface, since the prometheus emitter doesn't know when workers get garbage collected.
The following can also be handy:
- Concourse version: 3.13.0
- Deployment type (BOSH/Docker/binary): BOSH
- Infrastructure/IaaS: AWS
- Browser (if applicable): n/a
- Did this used to work? n/a