Hi folks

I hit an interesting error while trying to configure a public 3rd party OIDC provider via apiserver.config.k8s.io/v1beta1/AuthenticationConfiguration :

When configuring the apiserver to use a AuthenticationConfiguration that uses a issuer URL which has a public-trusted certificate...

--- 
apiVersion: v1
kind: ConfigMap
metadata:
  name: tenant-apiserver-authentication-config
data:
  authentication-configuration.yml: |
    ---
    apiVersion: apiserver.config.k8s.io/v1beta1
    kind: AuthenticationConfiguration
    jwt:
    - issuer:
        url: https://accounts.google.com/.well-known/openid-configuration
        audiences:
        - my-client
      claimMappings:
        username:
          claim: sub
          prefix: 'oidc:'
        uid:
          claim: sub
---
# relevant pieces of the tenant control plane:
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: tenant
spec:
...
  controlPlane:
    deployment:
...
      additionalVolumeMounts:
        apiServer:
        - mountPath: /etc/kubernetes/authentication-configuration.yml
          name: authentication-config
          readOnly: true
          subPath: authentication-configuration.yml
      additionalVolumes:
      - configMap:
          defaultMode: 420
          name: tenant-apiserver-authentication-config
        name: authentication-config
      extraArgs:
        apiServer:
        - --authentication-config=/etc/kubernetes/authentication-configuration.yml

then the kube-apiserver does not trust the certificate of https://accounts.google.com/.well-known/openid-configuration:

tenant-8c457c445-46vnb kube-apiserver E0126 08:42:52.572231       1 oidc.go:383] oidc authenticator: initializing plugin: Get "https://accounts.google.com/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by
unknown authority

In AuthenticationConfiguration, there is the option to specify a certifiacteAuthority, but it states that it falls back to use the system verifier (which is what we want here - to use the hosts CA bundle which contains the WebPKI roots):

# PEM encoded CA certificates used to validate the connection when fetching
# discovery information. If not set, **the system verifier will be used.**
# Same value as the content of the file referenced by the --oidc-ca-file flag.
certificateAuthority: <PEM encoded CA certificates>   

(from https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens)

While digging a bit into this, I found that kamaji replaces the system CAs of the kubernetes control plane pods with the CA trust bundle that's used within Kubernetes:

kubectl get deployment tenant -o yaml
...
  volumes:
...
      - name: etc-ssl-certs
        secret:
          defaultMode: 420
          secretName: tenant-ca
...

     volumeMounts:
...
        - mountPath: /etc/ssl/certs
          name: etc-ssl-certs
          readOnly: true
...

This essentially replaces the pods system CAs with the CA used for Kubernetes itself, preventing TLS connection to all systems outside this trust domain (for example, OIDC issuers, webhook etc.). I am not sure if I hit a bug or a feature :)

When looking at at usual kubeadm cluster, I see that they don't overwrite /etc/ssl/certs but just put the kube certs into /etc/kubernetes/pki:

# /etc/kubernetes/manifests/kube-apiserver.yaml from a kubeadm cluster
...
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
...
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
...

I think this setup would be nice - having the docker images system CA certs in /etc/ssl/certs and the kube CA next to it (for example in /etc/kubernetes/pki), where the kubernetes components explicitly reference it.

What do you think? Is this a bug? Or is there another reason why the Kubernetes CA became the system CA in kamaji?

My gut feeling would be to propose removing the etc-ssl-certs mount in kamaji, so that external systems can be used. If I remove that mount (manually), my connection works. But I am not sure if there is any other part of the control plane that relies on having the Kubernetes CA in the system truststore...