cert-manager in version 1.9.x and above is not able to log into HCP Vault with approle and a namespace #5563
Description
Describe the bug:
We use cert-manager on an AKS cluster (v1.23.12). cert-manager connects to a vault (v1.12.0) in the Hashicorp Cloud Platform.
Up to version 1.7.3 the cert-manager can log on to Vault and retrieve certificates without problems. With versions 1.9.x and higher this fails. Also in version 1.10.x I have no success. The configuration remains the same. After a downgrade to 1.7.3. the error is no longer present.
clusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: vault-issuer
namespace: platform
spec:
vault:
auth:
appRole:
path: approle
roleId: aaaabbbbccccddddeeeeeffffff9999
secretRef:
key: secretId
name: vault-approle
namespace: admin/platform
path: pki_xxxx/sign/xxxx-yyy-zzz
server: https://platform-public-vault-xxxxxxxxxxxxxxxx.yyyyyy.hashicorp.cloud:8200
It ends up with an error
Failed to initialize Vault client: error logging in to Vault server: Error making API request. URL: POST https://platform-public-vault-xxxxxxxxxxxxxxxx.yyyyyy.hashicorp.cloud:8200/v1/auth/approle/login Code: 403. Errors: * permission denied
I1107 13:21:37.246743 1 setup.go:108] vault-issuer-metrics: Failed to initialize Vault client: error logging in to Vault server: Error making API request.
URL: POST https://platform-public-vault-xxxxxxxxxxxxxxxx.yyyyyy.hashicorp.cloud:8200/v1/auth/approle/login
Code: 403. Errors:
* permission denied
E1107 13:21:37.246982 1 sync.go:62] cert-manager/clusterissuers "msg"="error setting up issuer" "error"="error logging in to Vault server: Error making API request.\n\nURL: POST https://platform-public-vault-xxxxxxxxxxxxxxxx.yyyyyy.hashicorp.cloud:8200/v1/auth/approle/login\nCode: 403. Errors:\n\n* permission denied" "resource_kind"="ClusterIssuer" "resource_name"="vault-issuer-metrics" "resource_namespace"="" "resource_version"="v1"
E1107 13:21:37.247175 1 controller.go:167] cert-manager/clusterissuers "msg"="re-queuing item due to error processing" "error"="error logging in to Vault server: Error making API request.\n\nURL: POST https://platform-public-vault-xxxxxxxxxxxxxxxx.yyyyyy.hashicorp.cloud:8200/v1/auth/approle/login\nCode: 403. Errors:\n\n* permission denied" "key"="vault-issuer-metrics"
I1107 13:21:37.247366 1 logs.go:177] cert-manager/controller "msg"="Event(v1.ObjectReference{Kind:\"ClusterIssuer\", Namespace:\"\", Name:\"vault-issuer-metrics\", UID:\"dcca815e-be28-48e9-97d9-e6d55075592d\", APIVersion:\"cert-manager.io/v1\", ResourceVersion:\"6054054\", FieldPath:\"\"}): type: 'Warning' reason: 'ErrInitIssuer' Error initializing issuer: error logging in to Vault server: Error making API request.\n\nURL: POST https://platform-public-vault-xxxxxxxxxxxxxxxx.yyyyyy.hashicorp.cloud:8200/v1/auth/approle/login\nCode: 403. Errors:\n\n* permission denied"
For me it looks like, the spec.vault.namespace value is not used anymore or the header in api call "X-VAULT-NAMESPACE:" is gone
After rolling back to 1.7.3, everything works as expected.
Expected behaviour:
getting cert and being able to login.
Steps to reproduce the bug:
create a namespace inside Vault
create approle role under that namespace
set up vault as an issuer following this
Afterwards update cert-manager from 1.7.3 to 1.9.1 with given config.
Environment details::
- Kubernetes version: v1.23.12
- Cloud-provider/provisioner: Azure
- cert-manager version: 1.7.2, 1.9.1, 1.10.0
- Install method: argocd -> helm
/kind bug