Exhausted container limit, contradicting container counts, container found for deletion constantly increasing #4340
Description
Bug Report
This is an uber-issue (3 issues in 1) since it's been snowballing since at least 2017, specifically #1669. The issue was resolved in some Concourse versions, but it is back with a vengeance since at least v5.4.0.
fly containers is reporting one number of containers (329), fly workers is reporting another (500). Which one is right? Since no more jobs can be scheduled, I would say fly workers. Attaching both files for inspection.
fly-workers.txt
fly-containers.txt
Lastly, the number of containers found for deletion is constantly increasing, it only resets itself when we delete workers.
Steps to Reproduce
Not sure. Container counts are off, containers are not GC'ed as documented, Caching & Retention is TODO.
Expected Results
I would expect containers to eventually be garbage collected.
Actual Results
Additional Info
Version Info
- Concourse version: 5.4.1
- Deployment type: BOSH
- Infrastructure/IaaS: GCP
- Did this used to work? Yes, worked fine in v5.3 and prior v5.x versions, stopped working after 5.4.0