Skip to content

Conversation

marseel
Copy link
Contributor

@marseel marseel commented May 26, 2025

In some of the runs, mnt directory is full and as a result, build fails. Usually, we were using up to ~15 GB of disk space on mnt directory, but default root has plenty of available disk space ~50 GB.

Related: #39682

Example run: https://github.com/cilium/cilium/actions/runs/15252030730/job/42890953705
Before even building images:

# Overview
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        72G   17G   55G  24% /
tmpfs           7.9G   84K  7.9G   1% /dev/shm
tmpfs           3.2G  1.1M  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdb16      881M   60M  760M   8% /boot
/dev/sdb15      105M  6.2M   99M   6% /boot/efi
/dev/sda1        74G   73G     0 100% /mnt
tmpfs           1.6G   12K  1.6G   1% /run/user/1001
ci: stop using /mnt directory for building images

In some of the runs, mnt directory is full and as a result, build fails.
Usually, we were using up to ~15 GB of disk space on mnt directory, but
default root has plenty of available disk space ~50 GB.

Related: #39682

Signed-off-by: Marcel Zieba <marcel.zieba@isovalent.com>
@marseel marseel added the release-note/ci This PR makes changes to the CI. label May 26, 2025
@marseel
Copy link
Contributor Author

marseel commented May 26, 2025

/test

@marseel marseel marked this pull request as ready for review May 26, 2025 12:57
@marseel marseel requested review from a team as code owners May 26, 2025 12:57
@aanm aanm enabled auto-merge May 26, 2025 14:05
@aanm aanm added this pull request to the merge queue May 26, 2025
Merged via the queue into main with commit e90242e May 26, 2025
77 checks passed
@aanm aanm deleted the pr/marseel/build_images_from_root_dir branch May 26, 2025 14:17
@joestringer
Copy link
Member

I guess if we find that this consistently solves GH worker disk space issues, then we might also want to backport it to stable branches?

I don't recall specifically seeing the disk space issue with /mnt on stable branches, but given that it occurs randomly based on the workers provided to us from GitHub, it seems like the same issue could impact stable branches.

@marseel
Copy link
Contributor Author

marseel commented May 28, 2025

I guess if we find that this consistently solves GH worker disk space issues, then we might also want to backport it to stable branches?

yup, I wanted to leave original issue open to double-check it and either backport or find alternative solution if it doesn't solve our problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note/ci This PR makes changes to the CI.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants