Cert-manager cannot be restricted to a namespace with --namespace #7676
Description
Describe the bug:
The controller CLI reference documents parameter --namespace string
If set, this limits the scope of cert-manager to a single namespace and ClusterIssuers are disabled. If not specified, all namespaces will be watched
Even when using --namespace=<namespace> cert-manager still attempts to access cluster-wide resources. As a result, it fails to operate correctly if deployed without cluster-wide RBAC permissions.
This issue was previously reported in #5524 but remains unresolved.
Expected behaviour:
Cert-manager should be deployable with RBAC permissions restricted to <namespace> when --namespace=<namespace> flag is used.
Steps to reproduce the bug:
As a quick hack, following can be used to deploy cert-manager as namespaced in a kind cluster
$ cd cert-manager
# See attachment in this issue for the .patch file
$ patch -p1 < e2e-setup.mk.patch.txt
# Change ClusterRoles and ClusterRoleBindings to Roles and RoleBindings
# Note: this does not make perfect modifications but it is good enough for testing
$ sed -i '
s/kind: ClusterRole$/kind: Role/;
s/kind: ClusterRoleBinding$/kind: RoleBinding/;
/metadata:/!b;n;/namespace:/!a\ namespace: {{ include "cert-manager.namespace" . }}
/roleRef:/!b;n;s/kind: ClusterRole/kind: Role/
' deploy/charts/cert-manager/templates/rbac.yaml
# Do not deploy webhooks
$ rm deploy/charts/cert-manager/templates/webhook-*
# Start kind cluster and deploy
$ make e2e-setup-certmanagerFollowing error gets repeatedly printed:
W0411 17:31:32.444733 1 reflector.go:569] k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot list resource "clusterissuers" in API group "cert-manager.io" at the cluster scope
E0411 17:31:32.444750 1 reflector.go:166] "Unhandled Error" err="k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *v1.ClusterIssuer: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User \"system:serviceaccount:cert-manager:cert-manager\" cannot list resource \"clusterissuers\" in API group \"cert-manager.io\" at the cluster scope" logger="UnhandledError"
full logs
I0411 17:23:46.689496 1 controller.go:292] "configured acme dns01 nameservers" logger="cert-manager.controller.build-context" nameservers=["10.0.0.16:53"]
I0411 17:23:46.689717 1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0411 17:23:46.689724 1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0411 17:23:46.689727 1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0411 17:23:46.689729 1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0411 17:23:46.694838 1 options.go:262] "enabling the sig-network Gateway API certificate-shim and HTTP-01 solver" logger="cert-manager"
I0411 17:23:46.694866 1 controller.go:89] "enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers gateway-shim ingress-shim issuers orders]" logger="cert-manager.controller"
I0411 17:23:46.694878 1 controller.go:444] "serving insecurely as tls certificate data not provided" logger="cert-manager.controller"
I0411 17:23:46.694886 1 controller.go:102] "listening for insecure connections" logger="cert-manager.controller" address="0.0.0.0:9402"
I0411 17:23:46.695143 1 controller.go:127] "starting metrics server" logger="cert-manager.controller" address="[::]:9402"
I0411 17:23:46.695161 1 controller.go:178] "starting leader election" logger="cert-manager.controller"
I0411 17:23:46.695181 1 controller.go:171] "starting healthz server" logger="cert-manager.controller" address="[::]:9403"
I0411 17:23:46.696802 1 leaderelection.go:257] attempting to acquire leader lease cert-manager/cert-manager-controller...
I0411 17:23:46.698975 1 leaderelection.go:271] successfully acquired lease cert-manager/cert-manager-controller
I0411 17:23:46.699061 1 controller.go:225] "skipping disabled controller" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-acme"
I0411 17:23:46.699070 1 controller.go:225] "skipping disabled controller" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-selfsigned"
I0411 17:23:46.700926 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-ca"
I0411 17:23:46.703130 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-selfsigned"
I0411 17:23:46.705409 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="ingress-shim"
I0411 17:23:46.707248 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-issuing"
I0411 17:23:46.708875 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-metrics"
I0411 17:23:46.711305 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="orders"
I0411 17:23:46.712949 1 controller.go:225] "skipping disabled controller" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-ca"
I0411 17:23:46.712990 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-venafi"
I0411 17:23:46.714833 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="gateway-shim"
I0411 17:23:46.716383 1 controller.go:225] "skipping disabled controller" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-vault"
I0411 17:23:46.716401 1 controller.go:225] "skipping disabled controller" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-venafi"
I0411 17:23:46.716409 1 controller.go:231] "skipping as cert-manager is scoped to a single namespace" logger="cert-manager.controller" controller="clusterissuers"
I0411 17:23:46.716508 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-key-manager"
I0411 17:23:46.718425 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="issuers"
I0411 17:23:46.720120 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-acme"
I0411 17:23:46.723291 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-approver"
I0411 17:23:46.724805 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-readiness"
I0411 17:23:46.726203 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-revision-manager"
I0411 17:23:46.727843 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-trigger"
I0411 17:23:46.729592 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="challenges"
I0411 17:23:46.731868 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-request-manager"
I0411 17:23:46.733387 1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-vault"
I0411 17:23:46.735032 1 reflector.go:376] Caches populated for *v1.Ingress from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.735144 1 reflector.go:376] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.735311 1 reflector.go:376] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.735325 1 reflector.go:376] Caches populated for *v1.Secret from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.735648 1 reflector.go:376] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
W0411 17:23:46.736096 1 reflector.go:569] k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot list resource "clusterissuers" in API group "cert-manager.io" at the cluster scope
E0411 17:23:46.736134 1 reflector.go:166] "Unhandled Error" err="k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *v1.ClusterIssuer: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User \"system:serviceaccount:cert-manager:cert-manager\" cannot list resource \"clusterissuers\" in API group \"cert-manager.io\" at the cluster scope" logger="UnhandledError"
I0411 17:23:46.742177 1 reflector.go:376] Caches populated for *v1.Certificate from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.794261 1 reflector.go:376] Caches populated for *v1.HTTPRoute from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.805607 1 reflector.go:376] Caches populated for *v1.Issuer from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.818865 1 reflector.go:376] Caches populated for *v1.Challenge from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.840145 1 reflector.go:376] Caches populated for *v1.Gateway from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.845608 1 reflector.go:376] Caches populated for *v1.CertificateRequest from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
I0411 17:23:46.850450 1 reflector.go:376] Caches populated for *v1.Order from k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251
W0411 17:23:47.851672 1 reflector.go:569] k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot list resource "clusterissuers" in API group "cert-manager.io" at the cluster scope
E0411 17:23:47.851699 1 reflector.go:166] "Unhandled Error" err="k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *v1.ClusterIssuer: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User \"system:serviceaccount:cert-manager:cert-manager\" cannot list resource \"clusterissuers\" in API group \"cert-manager.io\" at the cluster scope" logger="UnhandledError"
W0411 17:23:50.875330 1 reflector.go:569] k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot list resource "clusterissuers" in API group "cert-manager.io" at the cluster scope
E0411 17:23:50.875418 1 reflector.go:166] "Unhandled Error" err="k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *v1.ClusterIssuer: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User \"system:serviceaccount:cert-manager:cert-manager\" cannot list resource \"clusterissuers\" in API group \"cert-manager.io\" at the cluster scope" logger="UnhandledError"
W0411 17:23:55.118646 1 reflector.go:569] k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot list resource "clusterissuers" in API group "cert-manager.io" at the cluster scope
E0411 17:23:55.118762 1 reflector.go:166] "Unhandled Error" err="k8s.io/client-go@v0.32.0/tools/cache/reflector.go:251: Failed to watch *v1.ClusterIssuer: failed to list *v1.ClusterIssuer: clusterissuers.cert-manager.io is forbidden: User \"system:serviceaccount:cert-manager:cert-manager\" cannot list resource \"clusterissuers\" in API group \"cert-manager.io\" at the cluster scope" logger="UnhandledError"
...
Anything else we need to know?:
Including --controller=--controllers=*,-clusterissuers still results in the same error. However, using --controllers=*,-clusterissuers,-certificaterequests-issuer-selfsigned like suggested in #5524 (comment) eliminates the error but causes even namespaced self-signed issuers to stop working.
Environment details:
- Kubernetes version:
- Cloud-provider/provisioner:
- cert-manager version:
main - Install method: e.g. helm/static manifests
/kind bug