Skip to content

Conversation

cevich
Copy link
Member

@cevich cevich commented Jun 26, 2024

Previously a very complex, manual, and failure-prone import_images stage was required to bring raw images into EC2. Primarily this was necessary because beta images aren't published on EC2 by the fedoraproject. However, since the original implementation, CI operations against rawhide have largely supplanted the need to support testing against the beta images. This means the 'import_images' stage can be completely dropped, and the 'base_images' stage can simply source images (including rawhide if necessary) published by the Fedora project.

  • Test images in podman CI to confirm Debian pasta apparmor fix

@cevich cevich added no_skopeo_cidev Don't build the skopeo_cidev image no_prior-fedora Don't build any prior-fedora images no_fedora Don't build any fedora images no_build-push Don't build the build-push image no_fedora_podman Don't build the fedora_podman image no_prior-fedora_podman Don't build the prior-fedora_podman image no_fedora-netavark Don't build the fedora-netavark image no_fedora-podman-py Don't build the fedora-podman-py image no_image-builder Don't build the image-builder image no_win-server-wsl Don't build Windows WSL image no_debian Don't build the debian image no_rawhide Don't build the rawhide image labels Jun 26, 2024
@cevich cevich force-pushed the rm_import_images branch 2 times, most recently from b1cdd8d to 9863ea5 Compare June 26, 2024 18:50
@cevich
Copy link
Member Author

cevich commented Jun 26, 2024

Looks like this works for x86_64 and arm64. Rawhide support in EC2 is a possibility as well (though not currently needed). Leaving this as a draft until the big question can be answered: "Do/Will we ever need Beta Fedora CI VMs in EC2"?

The alternative is migrating the import_images stage to use Packer for only a small reduction in complexity.

@edsantiago
Copy link
Member

If I'm understanding this PR correctly, don't you also need to remove all the IMPORT_IMG_SFX and import_images stuff from Makefile?

@cevich
Copy link
Member Author

cevich commented Jun 27, 2024

don't you also need to remove all the

Oh, I thought I had. Maybe I missed committing it or something. In either case, my main quandary is knowing if we really will never again need to incorporate beta CI VM images. If so, I can move this simplification forward. If not, then I need to take the work for my Jira card in another direction. Perhaps this is something I should bring to the team.

@cevich cevich force-pushed the rm_import_images branch from 9863ea5 to 2d7ef74 Compare June 27, 2024 21:22
@cevich
Copy link
Member Author

cevich commented Jun 27, 2024

Force-push: Commit almost lost Makefile changes.

@edsantiago
Copy link
Member

Once again, I recommend .git/hooks/pre-push:

#!/bin/bash
#
# 2024-01-25 esm
#

# FIXME: figure out if this is a PR push or a tag push
#
imgsfx=$(<IMG_SFX)

imgsfx_history=.imgsfx.history
if [[ -e $imgsfx_history ]]; then
    if grep -q "$imgsfx" $imgsfx_history; then
        echo "FATAL: $imgsfx has already been used" >&2
        echo "Please rerun 'make IMG_SFX'" >&2
        exit 1
    fi
fi

echo $imgsfx >>$imgsfx_history

Copy link
Member

@edsantiago edsantiago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are still four IMPORT_IMG_SFX lines in Makefile that need removing

@cevich
Copy link
Member Author

cevich commented Jun 28, 2024

Sorry for not being clear, this is an experimental PR. It's far from perfect on purpose, I just wanted to see if it was possible to pickup the fedoraproject published AMIs.


You're pre-push hook is a neat hack, I wonder if it could be made to work with pre-commit (the app)? Either way, it would be nice if it were in the repo. (and README) so everyone knows to use it. I had completely forgotten about it.

@cevich
Copy link
Member Author

cevich commented Jun 28, 2024

I wonder if it could be made to work with pre-commit

Oh! Actually that's not very hard at all, just a bit of yaml and some minor script changes. Opened #364

@cevich cevich removed no_skopeo_cidev Don't build the skopeo_cidev image no_prior-fedora Don't build any prior-fedora images no_fedora Don't build any fedora images no_build-push Don't build the build-push image no_fedora_podman Don't build the fedora_podman image no_prior-fedora_podman Don't build the prior-fedora_podman image no_fedora-netavark Don't build the fedora-netavark image labels Jul 1, 2024
@cevich cevich removed no_fedora-podman-py Don't build the fedora-podman-py image no_image-builder Don't build the image-builder image no_win-server-wsl Don't build Windows WSL image no_debian Don't build the debian image no_rawhide Don't build the rawhide image labels Jul 1, 2024
cevich added 2 commits July 1, 2024 11:52
Signed-off-by: Chris Evich <cevich@redhat.com>
Previously a very complex, manual, and failure-prone `import_images`
stage was required to bring raw images into EC2.  Primarily this was
necessary because beta images aren't published on EC2 by the
fedoraproject.  However, since the original implementation, CI
operations against rawhide have largely supplanted the need to support
testing against the beta images.  This means the 'import_images' stage
can be completely dropped, and the 'base_images' stage can simply source
images (including `rawhide` if necessary) published by the Fedora
project.

Signed-off-by: Chris Evich <cevich@redhat.com>
@cevich cevich force-pushed the rm_import_images branch from 2d7ef74 to 2901478 Compare July 1, 2024 15:52
Copy link

github-actions bot commented Jul 1, 2024

Cirrus CI build successful. Found built image names and IDs:

Stage Image Name IMAGE_SUFFIX
base debian do-not-use
base fedora do-not-use
base fedora-aws do-not-use
base fedora-aws-arm64 do-not-use
base image-builder do-not-use
base prior-fedora do-not-use
cache build-push c20240701t155130z-f40f39d13
cache debian c20240701t155130z-f40f39d13
cache fedora c20240701t155130z-f40f39d13
cache fedora-aws c20240701t155130z-f40f39d13
cache fedora-netavark c20240701t155130z-f40f39d13
cache fedora-netavark-aws-arm64 c20240701t155130z-f40f39d13
cache fedora-podman-aws-arm64 c20240701t155130z-f40f39d13
cache fedora-podman-py c20240701t155130z-f40f39d13
cache prior-fedora c20240701t155130z-f40f39d13
cache rawhide c20240701t155130z-f40f39d13
cache win-server-wsl c20240701t155130z-f40f39d13

@edsantiago
Copy link
Member

debian prior-fedora fedora fedora-aws rawhide
kernel 6.9.7-1 6.9.6-100 6.9.7-200 6.9.6-200 6.8.5-301
6.8.12-1 ⇑ 6.9.4-100 ⇑ 6.9.5-200 ⇑ 6.9.4-200 ⇑
grub2-common 2.12-2 2.06-121 2.06-123 2.06-123 2.06-123
2.06-120 ⇑
aardvark-dns 1.4.0-6 1.11.0-1 1.11.0-1 1.11.0-1 1.11.0-1
1.4.0-5.1 ⇑
buildah 1.33.7+ds1-1 1.36.0-1 1.36.0-1 1.36.0-1 1.36.0-1
1.35.4-1 ⇑ 1.35.4-1 ⇑
container-selinux ? 2.232.1-1 2.232.1-1 2.232.1-1 2.232.1-1
2.231.0-1 ⇑
docker-ce 5:27.0.3-1~debian.12~bookworm ? ? ? ?
5:26.1.4-1~debian.12~bookworm ⇑
golang 2:1.22~3 1.21.11-1 1.22.4-1 1.22.4-1 1.22.4-4
1.22.4-3 ⇑
passt 2024-06-24 2024-06-07 2024-06-24 2024-06-24 2024-06-24
2024-06-07 ⇑ 2024-05-10 ⇑ 2024-06-07 ⇑ 2024-06-07 ⇑ 2024-06-07 ⇑
systemd 256.1-2 254.13-1 255.8-1 255.8-1 256.1-3
256.1-1 ⇑ 255.7-1 ⇑ 255.7-1 ⇑

@cevich
Copy link
Member Author

cevich commented Jul 1, 2024

Great, thanks. Hopefully the new passt (pasta?) package will resolve the old apparmor problem. Giving it a try in containers/podman#23157

@edsantiago
Copy link
Member

@cevich is this ready to merge? We have a lot of auto-imgs PRs in flight, and I would very much like to get this in, then my local-registry PRs, then the Windows one.

@cevich
Copy link
Member Author

cevich commented Jul 2, 2024

is this ready to merge?

I guess? I s'pose containers/podman#23157 does show all the Debian tests passing. Since you feel strongly that the PM MacOS failure is a flake, I guess this can go in? I'll un-draft it and let you merge in the order you feel appropriate.

@cevich cevich marked this pull request as ready for review July 2, 2024 15:01
@edsantiago edsantiago merged commit 38e7c58 into containers:main Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants