We previously restored the most recent yarn cache if yarn.lock changed. The problem is that that cache includes all dependencies of the old lockfile. If we now install dependencies with the new lockfile, the new cache will include all dependencies of the old lockfile + all dependencies of the new lockfile. Basically, we retain all previous versions of dependencies in the cache.

Ideally we'd check-in yarn v2's offline cache so that we only ever cache the used dependencies. We would only ever have to download new dependencies wheras we now have to download all dependencies when yarn.lock changes.

Note that the yarn cache is mostly for normalizing network. Downloading CircleCI cache is more stable than downloading from the npm registry (and has a slight benefit since CircleCI cache is a single request).

This will increase pipeline duration when yarn.lock changes (~30s best case for all deps up to ~2m 00s) but reduces pipeline duration when it doesn't change (~2m 00s currently for yarn cache down to 16s best case). The number of changes to yarn.lock are far smaller than the number of other changes so this tradeoff should be worth it. Right now we're downloading 2500 MB of node_modules cache even though we only use 274 MB.