Make CI image signing reliable #41138
Description
I've seen more than an acceptable level of CI image signing errors during CI builds recently. We should investigate how to make this step more reliable, or maybe optionally skip it for CI builds if it is introducing too much load on the underlying service.
Example failure on a PR: https://github.com/cilium/cilium/actions/runs/16943844870/job/48019697243?pr=41135#step:24:1
cosign sign -y quay.io/cilium/docker-plugin-ci@sha256:c7a1b5dbb2b2b0f18aa7721eabc75d8fb7aa1577d4afc3ba2932cb28ddca0320
if [[ "success" != 'skipped' ]]; then
cosign sign -y quay.io/cilium/docker-plugin-ci@sha256:a1a2114fc96f111ba082c41626bad26e85ef7b7840fbc7faab6459bb47b18c2f
fi
if [[ "success" != 'skipped' ]]; then
cosign sign -y quay.io/cilium/docker-plugin-ci@sha256:c87e23cb15a6d8528606503f2d514895ba4dad61f0da22a76709a41d256fdc25
fi
shell: /usr/bin/bash -e {0}
env:
QUAY_ORGANIZATION: cilium
QUAY_ORGANIZATION_DEV: cilium
QUAY_CHARTS_ORGANIZATION_DEV: cilium-charts-dev
EGRESS_GATEWAY_HELM_VALUES: --helm-set=egressGateway.enabled=true
BGP_CONTROL_PLANE_HELM_VALUES: --helm-set=bgpControlPlane.enabled=true
CILIUM_CLI_RELEASE_REPO: cilium/cilium-cli
CILIUM_CLI_VERSION:
CILIUM_CLI_IMAGE_REPO: quay.io/cilium/cilium-cli-ci
CILIUM_CLI_SKIP_BUILD: true
CILIUM_CLI_CODE_OWNERS_PATHS: CODEOWNERS
CILIUM_CLI_EXCLUDE_OWNERS: @cilium/github-sec
PUSH_TO_DOCKER_HUB: true
GCP_PERF_RESULTS_BUCKET: gs://cilium-scale-results
CILIUM_RUNTIME_IMAGE_PREFIX: quay.io/cilium/
KIND_VERSION: v0.29.0
KIND_K8S_IMAGE: quay.io/cilium/kindest-node:v1.34.0-rc.1@sha256:c60d9e8ae62dfa5128537407005efd1ae7965673b200f4da2c362bc0998a0322
KIND_K8S_VERSION: v1.34.0-rc.1
CILIUM_RUNTIME_IMAGE: quay.io/cilium/cilium-runtime:faaac4f040a5b14270b7122c084aca19c370776c@sha256:e01e27f4c241eb37cc1fbc5e07ef9927f89a8a4cc8e06b61e70831997652aaf3
setting TUF refresh period to 24h0m0s
Generating ephemeral keys...
Retrieving signed certificate...
Successfully verified SCT...
The sigstore service, hosted by sigstore a Series of LF Projects, LLC, is provided pursuant to the Hosted Project Tools Terms of Use, available at [https://lfprojects.org/policies/hosted-project-tools-terms-of-use/.](https://lfprojects.org/policies/hosted-project-tools-terms-of-use/)
Note that if your submission includes personal data associated with this signed artifact, it will be part of an immutable record.
This may include the email address associated with the account with which you authenticate your contractual Agreement.
This information will be used for signing this artifact and will be stored in public transparency logs and cannot be removed later, and is subject to the Immutable Record notice at [https://lfprojects.org/policies/hosted-project-tools-immutable-records/.](https://lfprojects.org/policies/hosted-project-tools-immutable-records/)
By typing 'y', you attest that (1) you are not submitting the personal data of any other person; and (2) you understand and agree to the statement and the Agreement terms at the URLs listed above.
tlog entry created with index: 390237817
Pushing signature to: quay.io/cilium/docker-plugin-ci
setting TUF refresh period to 24h0m0s
Generating ephemeral keys...
Retrieving signed certificate...
Error: signing [quay.io/cilium/docker-plugin-ci@sha256:a1a2114fc96f111ba082c41626bad26e85ef7b7840fbc7faab6459bb47b18c2f]: getting signer: getting key from Fulcio: retrieving cert: client: Post "https://fulcio.sigstore.dev/api/v1/signingCert": dial tcp 34.36.164.164:443: i/o timeout
error during command execution: signing [quay.io/cilium/docker-plugin-ci@sha256:a1a2114fc96f111ba082c41626bad26e85ef7b7840fbc7faab6459bb47b18c2f]: getting signer: getting key from Fulcio: retrieving cert: client: Post "https://fulcio.sigstore.dev/api/v1/signingCert": dial tcp 34.36.164.164:443: i/o timeout
Error: Process completed with exit code 1.