[Troubleshoot]: Reconciler error StorageError: invalid object, Code: 4 #329
Description
Describe the expected outcome
Create certificates.
Describe the actual outcome
We are getting the following errors can someone point us what could be the possible root cause?
{
"level": "error",
"ts": "2024-07-11T20:58:35Z",
"msg": "Reconciler error",
"controller": "certificaterequest",
"controllerGroup": "cert-manager.io",
"controllerKind": "CertificateRequest",
"CertificateRequest": {
"name": "istio-csr-vx59z",
"namespace": "istio-system"
},
"namespace": "istio-system",
"name": "istio-csr-vx59z",
"reconcileID": "9bef08f4-dc04-4929-b582-800a054a2495",
"error": "Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-vx59z\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-vx59z, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: cdc694e7-89bb-4c87-8745-2898ec235b66, UID in object meta: ",
"stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"
}
This was fresh installation on a cluster we were on version 1.2.5 and even after updating to 1.2.7 the issue persists.
The pod starts fine and signs a couple of certificates and after a while it just keeps on erroring:
{"level":"info","ts":"2024-07-11T21:21:19Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/086c62b520dbc87632f7b1c70841c8a3","certificaterequest":{"name":"istio-csr-74cd2","namespace":"istio-system"}
{"level":"info","ts":"2024-07-11T21:21:22Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/ce46ac96b76b9b76ed85c6a1045b337e","certificaterequest":{"name":"istio-csr-xr6sr","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:25Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/acccf3f7b2a5d5720630549dbee4eebe","certificaterequest":{"name":"istio-csr-4sgdc","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:28Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/1363d551d5008673f78b328cbadc081a","certificaterequest":{"name":"istio-csr-tsqqb","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:32Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/63eed2d60abab75686790d8067528eef","certificaterequest":{"name":"istio-csr-w2nbq","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:35Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/a1c53cc8b8a777f70b1271cc4bb0550e","certificaterequest":{"name":"istio-csr-2hp4b","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:38Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/5668b6797cf7426045738499ee4d4611","certificaterequest":{"name":"istio-csr-pw2bc","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:41Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/4a809e692c639eba0af1d1cbb28bbeb4","certificaterequest":{"name":"istio-csr-jsz85","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:45Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/e52c16c5b59fc154c68c8002d6a2224d","certificaterequest":{"name":"istio-csr-fr6q4","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:48Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/ed2d4a085176ce0ad0916ab97905f295","certificaterequest":{"name":"istio-csr-mlqzn","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:51Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/8651e097d7e90c7408c1f93ddf20cea0","certificaterequest":{"name":"istio-csr-5lxdp","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:54Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/92508cd2360f0e5a822095296c26b36b","certificaterequest":{"name":"istio-csr-vtzh4","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:58Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/48501e5b439555d1f9161edff48fc42f","certificaterequest":{"name":"istio-csr-7ld4m","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:01Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/c859a2762b454de1f00562921970fcea","certificaterequest":{"name":"istio-csr-bwfj7","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:04Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/fbd889f73d30699dc6882e8669510566","certificaterequest":{"name":"istio-csr-65rck","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:07Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/2314a4e1d389fba753f778eca59e73d3","certificaterequest":{"name":"istio-csr-hctps","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:11Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/1c6ac1721ab5db1ed7a3afbb2c41e0c8","certificaterequest":{"name":"istio-csr-h6s5x","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:14Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/5bfb97f192ece6d7c787eca66cc384a1","certificaterequest":{"name":"istio-csr-j8jb7","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:17Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/0c66c4245dae37f31f740ffeddd659cd","certificaterequest":{"name":"istio-csr-w2kgd","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:20Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/09e080cef89aba42cf47d833e8ef17a4","certificaterequest":{"name":"istio-csr-9rlvj","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:23Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/38433e2520d2f78b6bc318217fabb61f","certificaterequest":{"name":"istio-csr-4nnjq","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:27Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/43e4d8bb3f3bb7b2ba043534fcd5e2de","certificaterequest":{"name":"istio-csr-wldbt","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:30Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/830dfa03537874f55018eff6593e3d41","certificaterequest":{"name":"istio-csr-s8m2h","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:33Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-s8m2h","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-s8m2h","reconcileID":"747887b4-eb48-4cd3-b7c7-31e9eec7a47d","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-s8m2h\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-s8m2h, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5a74bd58-cf52-428d-83d1-3c396cc5bfc1, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:33Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/bc12e5fd8524c9f9df7c5ca0daff3ca4","certificaterequest":{"name":"istio-csr-bbh8l","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:36Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-bbh8l","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-bbh8l","reconcileID":"a15f399d-d7a6-4235-9504-e00255c0a96b","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-bbh8l\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-bbh8l, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c897e717-de1a-42cd-8ee7-7465e06e1933, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:36Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/f2a9c272e41e64b0beee3fa701bcd267","certificaterequest":{"name":"istio-csr-rqcs6","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:39Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-rqcs6","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-rqcs6","reconcileID":"dbc0bf65-0047-42c7-b30e-777c59bca54b","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-rqcs6\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-rqcs6, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d2a5535c-5764-41dd-8a67-8e796e70cf74, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:40Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/9712d7fcc73159c23c8d30ee5c8dcf7e","certificaterequest":{"name":"istio-csr-jq58j","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:43Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-jq58j","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-jq58j","reconcileID":"1cdabc6c-fb1f-4d5a-9bb8-0ec25e709667","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-jq58j\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-jq58j, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 680d59a6-8d8e-4c46-86f7-8b408e862c1d, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:43Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/21740b5f5867aadcf64f5b9d5c187838","certificaterequest":{"name":"istio-csr-hjpgr","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:46Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-hjpgr","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-hjpgr","reconcileID":"98423457-bd5c-4f17-853a-bc89d3cf04b1","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-hjpgr\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-hjpgr, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f40d5db3-c9ae-4022-a6a1-e5c86daab72f, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:46Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/bdec6e1422a60a8eea370dd767c79b33","certificaterequest":{"name":"istio-csr-zkp97","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:49Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-zkp97","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-zkp97","reconcileID":"09fb9a4e-1982-41db-b544-f791fca60950","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-zkp97\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-zkp97, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 41abe8bb-196f-4f06-8b49-eb5978f4d81c, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:49Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/1a2c04b63b32917829eca0839ab4eb75","certificaterequest":{"name":"istio-csr-6kkm6","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:52Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-6kkm6","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-6kkm6","reconcileID":"5fe3288a-49be-4093-9895-fdd4dd867a04","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-6kkm6\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-6kkm6, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1d5ff2f2-7801-4151-b9bf-6169d94b0af2, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:52Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/4e1fe5e8945ff38e84cff0765a0899ce","certificaterequest":{"name":"istio-csr-d9wv4","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:56Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-d9wv4","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-d9wv4","reconcileID":"717028ce-a681-45be-8722-234838caa8eb","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-d9wv4\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-d9wv4, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 85c68e8f-b471-47ba-a478-f9213dbeb782, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:56Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/7b954a013cbac45f0f1237a4dff09beb","certificaterequest":{"name":"istio-csr-rlsf5","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:59Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-rlsf5","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-rlsf5","reconcileID":"02672e7e-b3b7-4a8e-b2fc-1bbb59b61d29","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-rlsf5\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-rlsf5, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5826ad83-8829-4bff-94ed-71263f76ca3d, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:59Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/19f0e74278d056dbf60a8b3a56f41c3c","certificaterequest":{"name":"istio-csr-h6jmp","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:02Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-h6jmp","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-h6jmp","reconcileID":"75f699c7-be0b-4892-8d28-ca330c5f2a8e","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-h6jmp\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-h6jmp, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f02725d5-0610-480d-97f6-a5ed0c0cb42f, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:02Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/7f3f336abd9f53c09d0b022e406c1abb","certificaterequest":{"name":"istio-csr-g8wnd","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:05Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-g8wnd","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-g8wnd","reconcileID":"eff95137-95f6-4eaf-9462-e7d41f9f5ca8","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-g8wnd\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-g8wnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e3ed7235-1a07-437e-aa84-85b6d73ca07f, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:05Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/5ce31dfdb5b268392f132c3ce12a107c","certificaterequest":{"name":"istio-csr-xf4cr","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:08Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-xf4cr","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-xf4cr","reconcileID":"44456ff2-be9a-4509-a93b-fa125c114ef5","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-xf4cr\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-xf4cr, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 8acb81d2-2e3e-42a6-9f42-6d0105650f78, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:09Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/49c841b505a4310e853768f86c7926d3","certificaterequest":{"name":"istio-csr-k45ld","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:12Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-k45ld","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-k45ld","reconcileID":"5aeb7130-603f-4d6d-9040-2b1c2b6bcc4d","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-k45ld\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-k45ld, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d3e38bcf-fb73-418a-962c-b20dd21fbc04, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:12Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/7c37a63465ce0a97f5195f59cb38cf90","certificaterequest":{"name":"istio-csr-b9rxv","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:15Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-b9rxv","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-b9rxv","reconcileID":"7e9df590-79fe-41a6-80b0-57b6b787b23b","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-b9rxv\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-b9rxv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e0a502c0-4c55-42f0-814d-a075edff07ec, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:15Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/c4733808d820d80b1cd8241ad9c0f737","certificaterequest":{"name":"istio-csr-vg6dm","namespace":"istio-system"}}
...
Are we being rate limited by the AWS?
Steps to reproduce
No response
Relevant log output
No response
Version
v1.2.7
Have you tried the following?
- Check the Troubleshooting section
- Search open issues
Category
Supported Workflow Broken
Severity
Severity 1