Skip to content

Conversation

joestringer
Copy link
Member

@joestringer joestringer commented May 20, 2025

We recently started experiencing reports that there's not enough disk space.
For example, on CI image build runs, I observed the following error
while building a docker container image for a PR in the merge queue:

Error: buildx failed with: ERROR: failed to solve:
ResourceExhausted: failed to compute cache key:
mount callback failed on /tmp/containerd-mount2956800340:
write /tmp/containerd-mount2956800340/usr/lib/x86_64-linux-gnu/libc.so.6:
no space left on device

Several workflows were already unconditionally calling the clean up
script as an early step during the corresponding set up job. Update the
remaining users of the script to also unconditionally run the script.

We recently started experiencing reports that there's not enough disk space.
For example, on CI image build runs, I observed the following error
while building a docker container image for a PR in the merge queue:

  Error: buildx failed with: ERROR: failed to solve:
  ResourceExhausted: failed to compute cache key:
  mount callback failed on /tmp/containerd-mount2956800340:
  write /tmp/containerd-mount2956800340/usr/lib/x86_64-linux-gnu/libc.so.6:
  no space left on device

Several workflows were already unconditionally calling the clean up
script as an early step during the corresponding set up job. Update the
remaining users of the script to also unconditionally run the script.

Signed-off-by: Joe Stringer <joe@cilium.io>
@joestringer joestringer requested a review from a team as a code owner May 20, 2025 15:48
@joestringer joestringer added the release-note/ci This PR makes changes to the CI. label May 20, 2025
@joestringer joestringer requested review from a team as code owners May 20, 2025 15:48
@joestringer joestringer added the release-note/ci This PR makes changes to the CI. label May 20, 2025
@joestringer joestringer requested a review from pchaigno May 20, 2025 15:48
@joestringer
Copy link
Member Author

/test

@joestringer joestringer added this pull request to the merge queue May 20, 2025
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label May 20, 2025
Merged via the queue into main with commit c8cb040 May 20, 2025
253 of 254 checks passed
@joestringer joestringer deleted the pr/joe/always-clean-ci-workers branch May 20, 2025 17:56
@julianwiedmann
Copy link
Member

@joestringer do you expect that we'll need backports for this as well?

@joestringer joestringer added backport/author The backport will be carried out by the author of the PR. needs-backport/1.15 needs-backport/1.16 This PR / issue needs backporting to the v1.16 branch needs-backport/1.17 This PR / issue needs backporting to the v1.17 branch labels May 21, 2025
@joestringer
Copy link
Member Author

I don't know. I can try to port it as far back as we maintain.

@joestringer
Copy link
Member Author

I thought this would fix #39682, but it did not fix that issue.

@joestringer
Copy link
Member Author

Given that this conflicts with the approach in #34711 and it's not actually fixing the current issue we have, I'm not sure it's the right thing to backport this patch at this time. cc @marseel

@joestringer joestringer removed backport/author The backport will be carried out by the author of the PR. needs-backport/1.15 needs-backport/1.16 This PR / issue needs backporting to the v1.16 branch needs-backport/1.17 This PR / issue needs backporting to the v1.17 branch labels May 23, 2025
@tklauser tklauser added the needs-backport/1.17 This PR / issue needs backporting to the v1.17 branch label Jun 3, 2025
@tklauser tklauser removed the needs-backport/1.17 This PR / issue needs backporting to the v1.17 branch label Jun 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/ci This PR makes changes to the CI.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants