Skip to content

ACME GCP CA - Error waiting for authorization #7685

@filipbnkd

Description

@filipbnkd

📢 This issue has been addressed in cert-manager 1.18.1: https://github.com/cert-manager/cert-manager/releases/tag/v1.18.1

ℹ Read the cert-manager 1.18 release-notes to learn more.

Describe the bug:

During certificate renewal process, after successful ACME DNS01 validation record propagation, error (timeout) occurs during waiting for authorization. Issue first occurred on it's own during automatic renewal process (given config worked before).

Logs with logLevel: 6:

I0414 12:15:04.222969       1 sync.go:174] "No action taken" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:04.222998       1 sync.go:69] "skipping updating resource as new status == existing status" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:04.222990       1 dns.go:246] "preparing to create Cloudflare provider" logger="cert-manager.controller.Present.solverForChallenge" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" domain="customer.dev.domain.com"
I0414 12:15:04.228064       1 dns.go:104] "presenting DNS01 challenge for domain" logger="cert-manager.controller.Present" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" domain="customer.dev.domain.com"
I0414 12:15:07.842054       1 sync.go:125] "Computing list of Challenge resources that need to exist to complete this Order" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:07.842093       1 util.go:137] "selecting solver due to match all selector and no previously selected solver" logger="cert-manager.controller.challengeSpecForAuthorization" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:07.842162       1 sync.go:133] "Determining if any challenge resources need to be created" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:07.842183       1 sync.go:138] "Determining if any challenge resources need to be cleaned up" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:07.842213       1 sync.go:174] "No action taken" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:07.842241       1 sync.go:69] "skipping updating resource as new status == existing status" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:10.811901       1 dns.go:118] "checking DNS propagation" logger="cert-manager.controller.Check" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" domain="customer.dev.domain.com" nameservers=["1.1.1.1:53","1.0.0.1:53"]
I0414 12:15:10.811986       1 logs.go:185] "Event(v1.ObjectReference{Kind:\"Challenge\", Namespace:\"development\", Name:\"customer-5-1296540651-3907418579\", UID:\"d85377e2-3e73-43e1-b831-11f0daca8a1a\", APIVersion:\"acme.cert-manager.io/v1\", ResourceVersion:\"1974292641\", FieldPath:\"\"}): type: 'Normal' reason: 'Presented' Presented challenge using DNS-01 challenge mechanism" logger="cert-manager.controller"
I0414 12:15:10.837182       1 wait.go:145] "Looking up TXT records" logger="cert-manager.controller" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" fqdn="_acme-challenge.customer.dev.domain.com."
E0414 12:15:10.837225       1 sync.go:208] "propagation check failed" err="DNS record for \"customer.dev.domain.com\" not yet propagated" logger="cert-manager.controller" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01"
I0414 12:15:10.852861       1 sync.go:125] "Computing list of Challenge resources that need to exist to complete this Order" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:10.852875       1 util.go:137] "selecting solver due to match all selector and no previously selected solver" logger="cert-manager.controller.challengeSpecForAuthorization" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:10.852916       1 round_trippers.go:560] PUT https://172.30.0.1:443/apis/acme.cert-manager.io/v1/namespaces/development/challenges/customer-5-1296540651-3907418579/status 200 OK in 15 milliseconds
I0414 12:15:10.852935       1 sync.go:133] "Determining if any challenge resources need to be created" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:10.852952       1 sync.go:138] "Determining if any challenge resources need to be cleaned up" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:10.852980       1 sync.go:174] "No action taken" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:10.853007       1 sync.go:69] "skipping updating resource as new status == existing status" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:15:10.854525       1 dns.go:118] "checking DNS propagation" logger="cert-manager.controller.Check" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" domain="customer.dev.domain.com" nameservers=["1.1.1.1:53","1.0.0.1:53"]
I0414 12:15:10.875424       1 wait.go:145] "Looking up TXT records" logger="cert-manager.controller" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" fqdn="_acme-challenge.customer.dev.domain.com."
E0414 12:15:10.875465       1 sync.go:208] "propagation check failed" err="DNS record for \"customer.dev.domain.com\" not yet propagated" logger="cert-manager.controller" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01"
I0414 12:15:20.838131       1 dns.go:118] "checking DNS propagation" logger="cert-manager.controller.Check" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" domain="customer.dev.domain.com" nameservers=["1.1.1.1:53","1.0.0.1:53"]
I0414 12:15:20.860225       1 wait.go:145] "Looking up TXT records" logger="cert-manager.controller" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" fqdn="_acme-challenge.customer.dev.domain.com."
I0414 12:15:20.870186       1 wait.go:145] "Looking up TXT records" logger="cert-manager.controller" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" fqdn="_acme-challenge.customer.dev.domain.com."
I0414 12:15:20.870218       1 wait.go:160] "Selfchecking using the DNS Lookup method was successful" logger="cert-manager.controller" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01"
I0414 12:15:20.870240       1 dns.go:130] "waiting DNS record TTL to allow the DNS01 record to propagate for domain" logger="cert-manager.controller.Check" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" domain="customer.dev.domain.com" ttl=60 fqdn="_acme-challenge.customer.dev.domain.com."
I0414 12:16:20.871164       1 dns.go:132] "ACME DNS01 validation record propagated" logger="cert-manager.controller.Check" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" domain="customer.dev.domain.com" fqdn="_acme-challenge.customer.dev.domain.com."
I0414 12:16:20.871243       1 sync.go:375] "accepting challenge with ACME server" logger="cert-manager.controller.acceptChallenge" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01"
I0414 12:16:21.125374       1 sync.go:392] "waiting for authorization for domain" logger="cert-manager.controller.acceptChallenge" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01"
E0414 12:16:41.125533       1 sync.go:403] "error waiting for authorization" err="context deadline exceeded" logger="cert-manager.controller.acceptChallenge" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01"
E0414 12:16:41.125638       1 sync.go:240] "unexpected non-ACME API error" err="context deadline exceeded" logger="cert-manager.controller" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01"
I0414 12:16:41.146380       1 round_trippers.go:560] PUT https://172.30.0.1:443/apis/acme.cert-manager.io/v1/namespaces/development/challenges/customer-5-1296540651-3907418579/status 200 OK in 20 milliseconds
I0414 12:16:41.146603       1 sync.go:125] "Computing list of Challenge resources that need to exist to complete this Order" logger="cert-manager.controller" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:16:41.146673       1 util.go:137] "selecting solver due to match all selector and no previously selected solver" logger="cert-manager.controller.challengeSpecForAuthorization" resource_name="customer-5-1296540651" resource_namespace="development" resource_kind="Order" resource_version="v1"
I0414 12:16:41.146763       1 dns.go:246] "preparing to create Cloudflare provider" logger="cert-manager.controller.CleanUp.solverForChallenge" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" dnsName="customer.dev.domain.com" type="DNS-01" resource_name="customer-5-1296540651-3907418579" resource_namespace="development" resource_kind="Challenge" resource_version="v1" domain="customer.dev.domain.com"
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: gcp
spec:
  acme:
    disableAccountKeyGeneration: true
    email: myemail@mydomain.com
    privateKeySecretRef:
      name: gcpca
    server: https://dv.acme-v02.api.pki.goog/directory
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            key: api-key
            name: cloudflare-api-cert-manager
          email: myemail@mydomain.com
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-customer
  namespace: development
spec:
  commonName: customer.dev.mydomain.com
  dnsNames:
  - customer.dev.mydomain.com
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: gcp
  secretName: ag-customer-web-cert

Expected behaviour:

Certificate should be renewed.

Steps to reproduce the bug:

Applying above mentioned resources and referenced secrets should be enough.

Anything else we need to know?:

Environment details:

  • Kubernetes version: 1.31.6-gke.1020000
  • Cloud-provider/provisioner: GCP/GKE
  • cert-manager version: v1.17.1
  • Install method: helm

/kind bug

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions