[BUG] Race condition when an existing artifact is being fetched right about when new files are added there #1235
Description
Describe the bug
Sometimes, when an artifact is being updated with new files, the artifact download fails because the API lists those new files as being a part of the artifact but the files themselves haven't yet been made available though the CDN.
The client iterates over the entries, expecting that the files are already downloadable, and crashes when it gets a 404.
See the additional context links to understand the code path that leads to this. But TL;DR — adding HttpCodes.NotFound to
toolkit/packages/artifact/src/internal/utils.ts
Lines 74 to 81 in b36e704
To Reproduce
Have a few sufficiently big CI matrixes depending on a job creating the initial artifact.
These matrixes should be downloading those artifacts and uploading new files there simultaneously.
You may need to do a number of restarts to hit a race condition when some of the steps that make additional files would add them close in time to the steps in other jobs making artifact downloads.
Expected behavior
Artifact file downloads should re-try on 404 Not Found.
Screenshots
N/A
Additional context