Feature Request

What challenge are you facing?

I'm using concourse 3.8 with the random container placement strategy. The random placement is necessary because we've had too much trouble with "worker gravity" overloading individual workers in our 6-node cluster. Obviously, the random worker placement resulted in decreased hit-rates for task caches.

A Modest Proposal

Task caches are currently local to a worker. I propose that concourse, after a successful build, asynchronously tries to transfer the cache volume using baggageclaim to a central "cache store".

When executing a task and workers fail to find a cache volume locally, they can try to pull one from the central "cache store". Since it's just caches, this behavior can be "bolted on". It's not important that workers retrieve the most recent version of a cache volume from the "cache store" under all circumstances (no locking etc. required), but more important that they receive any version at all.