kubernetes driver can't authorize with GitLab KAS #1884
Description
Contributing guidelines
- I've read the contributing guidelines and wholeheartedly agree
I've found a bug and checked that ...
- ... the documentation does not mention anything about my problem
- ... there are no open or closed issues that are related to my problem
Description
Using buildx with --driver=kubernetes using GitLab Kubernetes Agent Server fails Authorization.
Expected behaviour
Authorization should succeed given that kubectl commands within the same context perform as expected.
Actual behaviour
While using buildx with --driver=kubernetes from within GitLab pipelines attempting to utilize existing cluster I get:
#1 ERROR: error for bootstrap "cngbuilder-amd64": GitLab Agent Server: Unauthorized: no valid credentials provided. Trace ID: e505502d5b250ff193d441b6ee1b2fd0
#2 [cngbuilder-arm64 internal] booting buildkit
#2 ERROR: error for bootstrap "cngbuilder-arm64": GitLab Agent Server: Unauthorized: no valid credentials provided. Trace ID: 516b84bc883ea687ccd0c85479f82eaf
Buildx version
github.com/docker/buildx v0.10.5 86bdced
Docker info
No response
Builders list
$ docker buildx inspect --bootstrap
#1 [cngbuilder-amd64 internal] booting buildkit
#1 ERROR: error for bootstrap "cngbuilder-amd64": GitLab Agent Server: Unauthorized: no valid credentials provided. Trace ID: bbb6079378d897ce5ec4370d83e1dbd8
#2 [cngbuilder-arm64 internal] booting buildkit
#2 ERROR: error for bootstrap "cngbuilder-arm64": GitLab Agent Server: Unauthorized: no valid credentials provided. Trace ID: 778e870e2131f7509f90ea2f27c1f06e
------
> [cngbuilder-amd64 internal] booting buildkit:
------
------
> [cngbuilder-arm64 internal] booting buildkit:
------
Name: cngbuilder
Driver: kubernetes
Last Activity: 2023-06-14 20:31:30 +0000 UTC
Nodes:
Name: cngbuilder-amd64
Endpoint: kubernetes:///cngbuilder?deployment=cngbuilder-amd64&kubeconfig=%2Fbuilds%2Fgitlab-org%2Fbuild%2FCNG.tmp%2FKUBECONFIG
Driver Options: namespace="default" nodeselector="kubernetes.io/arch=amd64"
Status: inactive
Platforms: linux/amd64*
Name: cngbuilder-arm64
Endpoint: kubernetes:///cngbuilder?deployment=cngbuilder-arm64&kubeconfig=%2Fbuilds%2Fgitlab-org%2Fbuild%2FCNG.tmp%2FKUBECONFIG
Driver Options: namespace="default" nodeselector="kubernetes.io/arch=arm64"
Status: inactive
Platforms: linux/arm64*
Configuration
FROM alpine
RUN echo "hello"Build logs
No response
Additional info
Talking to GitLab KAS developers I've got this response:
no valid credentials providedmeans that neither did the request contain theAuthorizationheader (usually containing the CI Job Token), nor the session cookie (used via the new Environments Dashboard UI). (see code here)
so while it's difficult to troubleshoot immediately due to complexity of stack involved it does seem like there's a good chance buildx kubernetes driver strips portion of request resulting in above failures.
More on the subject here: https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/1368#note_1430995638 (including some logs etc.)