Unable to build operator image with CGO on arm64 #35324
Description
The cilium-operator docker image fails to build if CGO is enabled. CGO is triggered when building the operator with race detection enabled, which is specified by passing the MODIFIERS=RACE=1 build arg.
$ git rev-parse HEAD
81cd7d6ee2cef1491bca2c97ecc6de4d5dda97b9
$ docker buildx build \
--build-arg OPERATOR_VARIANT=operator-generic \
--build-arg MODIFIERS=RACE=1 \
--file ./images/operator/Dockerfile \
--platform linux/arm64 \
--target release \
--tag quay.io/learnitall/operator-generic-ci \
--progress=plain \
.
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 4.40kB done
#1 DONE 0.0s
#2 [internal] load metadata for docker.io/library/golang:1.23.2@sha256:adee809c2d0009a4199a11a1b2618990b244c6515149fe609e2788ddf164bd10
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/alpine:3.20.3@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d
#3 DONE 0.0s
#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s
#5 [internal] load build context
#5 DONE 0.0s
#6 [builder 1/4] FROM docker.io/library/golang:1.23.2@sha256:adee809c2d0009a4199a11a1b2618990b244c6515149fe609e2788ddf164bd10
#6 resolve docker.io/library/golang:1.23.2@sha256:adee809c2d0009a4199a11a1b2618990b244c6515149fe609e2788ddf164bd10 0.0s done
#6 DONE 0.0s
#7 [certs 1/2] FROM docker.io/library/alpine:3.20.3@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d
#7 resolve docker.io/library/alpine:3.20.3@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d 0.0s done
#7 DONE 0.0s
#8 [builder 2/4] WORKDIR /go/src/github.com/cilium/cilium
#8 CACHED
#5 [internal] load build context
#5 transferring context: 225.99MB 1.5s done
#5 DONE 1.6s
#9 [gops 2/4] WORKDIR /go/src/github.com/cilium/cilium/images/runtime
#9 CACHED
#10 [gops 3/4] RUN apt-get update && apt-get install -y binutils-aarch64-linux-gnu binutils-x86-64-linux-gnu
#10 CACHED
#11 [builder 3/4] RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium --mount=type=cache,target=/root/.cache --mount=type=cache,target=/go/pkg make GOARCH=arm64 DESTDIR=/out/linux/arm64 RACE=1 build-container-operator-generic install-container-bi
#11 0.375 make -C operator cilium-operator-generic
#11 0.378 make[1]: Entering directory '/go/src/github.com/cilium/cilium/operator'
#11 0.434 CGO_ENABLED=1 CC=aarch64-linux-gnu-gcc go build -mod=vendor -race -ldflags '-X "github.com/cilium/cilium/pkg/version.ciliumVersion=1.17.0-dev " -s -w -X "github.com/cilium/cilium/pkg/envoy.requiredEnvoyVersionSHA=e2dfcc576d5152c967479115e0e0a3905be766bb" ' -tags=osusergo,
#11 ...
#12 [gops 4/4] RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium --mount=type=cache,target=/root/.cache --mount=type=cache,target=/go/pkg ./build-gops.sh
#12 0.282 + set -o errexit
#12 0.282 + set -o pipefail
#12 0.282 + set -o nounset
#12 0.282 + gops_version=v0.3.27
#12 0.282 + mkdir -p /go/src/github.com/google
#12 0.283 + cd /go/src/github.com/google
#12 0.283 + git clone https://github.com/google/gops.git
#12 0.284 Cloning into 'gops'...
#12 2.252 + cd gops
#12 2.252 + git checkout -b v0.3.27 v0.3.27
#12 2.255 Switched to a new branch 'v0.3.27'
#12 2.255 + git --no-pager remote -v
#12 2.256 origin https://github.com/google/gops.git (fetch)
#12 2.256 origin https://github.com/google/gops.git (push)
#12 2.256 + git --no-pager log -1
#12 2.257 commit 70408a1556a9d904d8c79722aebe1aaa3b24beca
#12 2.257 Author: Tobias Klauser <tklauser@distanz.ch>
#12 2.257 Date: Fri Feb 10 14:18:59 2023 +0100
#12 2.257
#12 2.257 Update CI to Go 1.20
#12 2.257 + for arch in amd64 arm64
#12 2.257 + mkdir -p /out/linux/amd64/bin
#12 2.258 + GOARCH=amd64
#12 2.258 + CGO_ENABLED=0
#12 2.258 + go build -ldflags '-s -w' -o /out/linux/amd64/bin/gops github.com/google/gops
#12 2.387 + for arch in amd64 arm64
#12 2.387 + mkdir -p /out/linux/arm64/bin
#12 2.388 + GOARCH=arm64
#12 2.388 + CGO_ENABLED=0
#12 2.388 + go build -ldflags '-s -w' -o /out/linux/arm64/bin/gops github.com/google/gops
#12 2.527 + x86_64-linux-gnu-strip /out/linux/amd64/bin/gops
#12 2.541 + aarch64-linux-gnu-strip /out/linux/arm64/bin/gops
#12 DONE 2.8s
#11 [builder 3/4] RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium --mount=type=cache,target=/root/.cache --mount=type=cache,target=/go/pkg make GOARCH=arm64 DESTDIR=/out/linux/arm64 RACE=1 build-container-operator-generic install-container-binary-operator-generic
#11 3.215 # runtime/cgo
#11 3.215 cgo: C compiler "aarch64-linux-gnu-gcc" not found: exec: "aarch64-linux-gnu-gcc": executable file not found in $PATH
#11 10.32 make[1]: *** [Makefile:28: cilium-operator-generic] Error 1
#11 10.32 make[1]: Leaving directory '/go/src/github.com/cilium/cilium/operator'
#11 10.32 make: *** [Makefile:65: build-container-operator-generic] Error 2
#11 ERROR: process "/bin/sh -c make GOARCH=${TARGETARCH} DESTDIR=/out/${TARGETOS}/${TARGETARCH} ${MODIFIERS} build-container-${OPERATOR_VARIANT} install-container-binary-${OPERATOR_VARIANT}" did not complete successfully: exit code: 2
------
> [builder 3/4] RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium --mount=type=cache,target=/root/.cache --mount=type=cache,target=/go/pkg make GOARCH=arm64 DESTDIR=/out/linux/arm64 RACE=1 build-container-operator-generic install-container-binary-operator-generic:
0.375 make -C operator cilium-operator-generic
0.378 make[1]: Entering directory '/go/src/github.com/cilium/cilium/operator'
0.434 CGO_ENABLED=1 CC=aarch64-linux-gnu-gcc go build -mod=vendor -race -ldflags '-X "github.com/cilium/cilium/pkg/version.ciliumVersion=1.17.0-dev " -s -w -X "github.com/cilium/cilium/pkg/envoy.requiredEnvoyVersionSHA=e2dfcc576d5152c967479115e0e0a3905be766bb" ' -tags=osusergo,lockdebug,ipam_provider_operator -mod=vendor -race -ldflags '-X "github.com/cilium/cilium/pkg/version.ciliumVersion=1.17.0-dev " -s -w -X "github.com/cilium/cilium/pkg/envoy.requiredEnvoyVersionSHA=e2dfcc576d5152c967479115e0e0a3905be766bb" ' -tags=osusergo,lockdebug,ipam_provider_operator -o cilium-operator-generic
3.215 # runtime/cgo
3.215 cgo: C compiler "aarch64-linux-gnu-gcc" not found: exec: "aarch64-linux-gnu-gcc": executable file not found in $PATH
10.32 make[1]: *** [Makefile:28: cilium-operator-generic] Error 1
10.32 make[1]: Leaving directory '/go/src/github.com/cilium/cilium/operator'
10.32 make: *** [Makefile:65: build-container-operator-generic] Error 2
------
Dockerfile:25
--------------------
24 |
25 | >>> RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium \
26 | >>> --mount=type=cache,target=/root/.cache \
27 | >>> --mount=type=cache,target=/go/pkg \
28 | >>> make GOARCH=${TARGETARCH} DESTDIR=/out/${TARGETOS}/${TARGETARCH} ${MODIFIERS} \
29 | >>> build-container-${OPERATOR_VARIANT} install-container-binary-${OPERATOR_VARIANT}
30 |
--------------------
ERROR: failed to solve: process "/bin/sh -c make GOARCH=${TARGETARCH} DESTDIR=/out/${TARGETOS}/${TARGETARCH} ${MODIFIERS} build-container-${OPERATOR_VARIANT} install-container-binary-${OPERATOR_VARIANT}" did not complete successfully: exit code: 2
The error returned indicates that the arm64 compiler was not found in the builder image. The builder image in the cilium-operator Dockerfile is set to docker.io/library/golang with the platform matching the build platform:
cilium/images/operator/Dockerfile
Lines 1 to 12 in 81cd7d6
When running on an amd64 system, the golang image won't contain an arm64 compiler.
One solution is to use the Cilium builder image instead of the golang image. This seems to compile just fine, but I haven't tried testing it yet:
$ head -n 13 images/operator/Dockerfile
# Copyright Authors of Cilium
# SPDX-License-Identifier: Apache-2.0
ARG BASE_IMAGE=scratch
ARG GOLANG_IMAGE=docker.io/library/golang:1.23.2@sha256:adee809c2d0009a4199a11a1b2618990b244c6515149fe609e2788ddf164bd10
ARG ALPINE_IMAGE=docker.io/library/alpine:3.20.3@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d
ARG CILIUM_BUILDER_IMAGE=quay.io/cilium/cilium-builder:f5be233da8726ac46f5df9d6774be4c28063b058@sha256:4c310d92f42cbfd7f742c5c74396ddab18073aba238257212bbaf55fee1019e8
# BUILDPLATFORM is an automatic platform ARG enabled by Docker BuildKit.
# Represents the plataform where the build is happening, do not mix with
# TARGETARCH
FROM --platform=${BUILDPLATFORM} ${CILIUM_BUILDER_IMAGE} AS builder
$ docker buildx build \
--build-arg OPERATOR_VARIANT=operator-generic \
--build-arg MODIFIERS=RACE=1 \
--file ./images/operator/Dockerfile \
--platform linux/arm64 \
--target release \
--tag quay.io/learnitall/operator-generic-ci \
--progress=plain \
.
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 4.41kB done
#1 DONE 0.0s
#2 [internal] load metadata for docker.io/library/golang:1.23.2@sha256:adee809c2d0009a4199a11a1b2618990b244c6515149fe609e2788ddf164bd10
#2 DONE 0.1s
#3 [internal] load metadata for docker.io/library/alpine:3.20.3@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d
#3 DONE 0.1s
#4 [internal] load metadata for quay.io/cilium/cilium-builder:f5be233da8726ac46f5df9d6774be4c28063b058@sha256:4c310d92f42cbfd7f742c5c74396ddab18073aba238257212bbaf55fee1019e8
#4 DONE 0.1s
#5 [internal] load .dockerignore
#5 transferring context: 2B done
#5 DONE 0.0s
#6 [internal] load build context
#6 DONE 0.0s
#7 [certs 1/2] FROM docker.io/library/alpine:3.20.3@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d
#7 resolve docker.io/library/alpine:3.20.3@sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d 0.1s done
#7 DONE 0.1s
#8 [builder 1/4] FROM quay.io/cilium/cilium-builder:f5be233da8726ac46f5df9d6774be4c28063b058@sha256:4c310d92f42cbfd7f742c5c74396ddab18073aba238257212bbaf55fee1019e8
#8 resolve quay.io/cilium/cilium-builder:f5be233da8726ac46f5df9d6774be4c28063b058@sha256:4c310d92f42cbfd7f742c5c74396ddab18073aba238257212bbaf55fee1019e8 0.1s done
#8 DONE 0.1s
#9 [builder 2/4] WORKDIR /go/src/github.com/cilium/cilium
#9 CACHED
#10 [gops 1/4] FROM docker.io/library/golang:1.23.2@sha256:adee809c2d0009a4199a11a1b2618990b244c6515149fe609e2788ddf164bd10
#10 resolve docker.io/library/golang:1.23.2@sha256:adee809c2d0009a4199a11a1b2618990b244c6515149fe609e2788ddf164bd10 0.1s done
#10 DONE 0.1s
#6 [internal] load build context
#6 transferring context: 225.99MB 1.8s done
#6 DONE 1.9s
#11 [builder 3/4] RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium --mount=type=cache,target=/root/.cache --mount=type=cache,target=/go/pkg make GOARCH=arm64 DESTDIR=/out/linux/arm64 RACE=1 build-container-operator-generic install-container-binary-operator-generic
#11 0.502 make -C operator cilium-operator-generic
#11 0.506 make[1]: Entering directory '/go/src/github.com/cilium/cilium/operator'
#11 0.574 CGO_ENABLED=1 CC=aarch64-linux-gnu-gcc go build -mod=vendor -race -ldflags '-X "github.com/cilium/cilium/pkg/version.ciliumVersion=1.17.0-dev " -s -w -X "github.com/cilium/cilium/pkg/envoy.requiredEnvoyVersionSHA=e2dfcc576d5152c967479115e0e0a3905be766bb" ' -tags=osusergo,lockdebug,ipam_provider_operator -mod=vendor -race -ldflags '-X "github.com/cilium/cilium/pkg/version.ciliumVersion=1.17.0-dev " -s -w -X "github.com/cilium/cilium/pkg/envoy.requiredEnvoyVersionSHA=e2dfcc576d5152c967479115e0e0a3905be766bb" ' -tags=osusergo,lockdebug,ipam_provider_operator -o cilium-operator-generic
#11 21.26 make[1]: Leaving directory '/go/src/github.com/cilium/cilium/operator'
#11 21.26 install -m 0755 -d /out/linux/arm64/usr/bin
#11 21.26 make -C operator install-generic
#11 21.26 make[1]: Entering directory '/go/src/github.com/cilium/cilium/operator'
#11 21.29 install -m 0755 -d /out/linux/arm64/usr/bin
#11 21.30 install -m 0755 cilium-operator-generic /out/linux/arm64/usr/bin
#11 21.35 make[1]: Leaving directory '/go/src/github.com/cilium/cilium/operator'
#11 DONE 21.6s
#12 [builder 4/4] RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium --mount=type=cache,target=/root/.cache --mount=type=cache,target=/go/pkg make GOARCH=${BUILDARCH} licenses-all && mv LICENSE.all /out/linux/arm64
#12 DONE 0.6s
#13 [certs 2/2] RUN apk --update add ca-certificates
#13 CACHED
#14 [release 1/5] COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
#14 CACHED
#15 [gops 2/4] WORKDIR /go/src/github.com/cilium/cilium/images/runtime
#15 CACHED
#16 [gops 3/4] RUN apt-get update && apt-get install -y binutils-aarch64-linux-gnu binutils-x86-64-linux-gnu
#16 CACHED
#17 [gops 4/4] RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium --mount=type=cache,target=/root/.cache --mount=type=cache,target=/go/pkg ./build-gops.sh
#17 CACHED
#18 [release 2/5] COPY --from=gops /out/linux/arm64/bin/gops /bin/gops
#18 CACHED
#19 [release 3/5] COPY --from=builder /out/linux/arm64/usr/bin/cilium-operator-generic /usr/bin/cilium-operator-generic
#19 DONE 0.1s
#20 [release 4/5] COPY --from=builder /out/linux/arm64/LICENSE.all /LICENSE.all
#20 DONE 0.0s
#21 exporting to image
#21 exporting layers
#21 exporting layers 3.6s done
#21 exporting manifest sha256:49da9763f22c7f54432c19dcb87c54e1e4a812b6a9738e663c3f489ebd90b745 0.0s done
#21 exporting config sha256:b95e8e3bd6db4450a54135aebebb8b8d32d4d9b9f4f55e0390da40d8805672be done
#21 exporting attestation manifest sha256:fe1d2b4a91c5d9c2c0dd0d6e5962e4c45eb6da8e3e836d214b72c92df11e5c04 0.0s done
#21 exporting manifest list sha256:f70d81f446ad48f5bbc9d65fb6fe062b18dee4308bfc9166f139bb32659e83df done
#21 naming to quay.io/learnitall/operator-generic-ci:latest done
#21 DONE 3.7s