Skip to content

CI: ConformanceKind1.19: Error: Unable to enable Hubble: timeout while waiting for status to become successful: context deadline exceeded #25940

@YutaroHayakawa

Description

@YutaroHayakawa

CI failure

Observed in the v1.11 backport PR (https://github.com/cilium/cilium/actions/runs/5185762997/jobs/9346045554?pr=25920)

Sysdump: cilium-sysdump-out.zip.zip

🔑 Found CA in secret cilium-ca
ℹ️  helm template --namespace kube-system cilium "install/kubernetes/cilium" --version 1.11.17 --set cluster.id=0,cluster.name=kind-chart-testing,clustermesh.apiserver.image.repository=quay.io/cilium/clustermesh-apiserver-ci,clustermesh.apiserver.image.tag=6d5f174e837a7a4afbc3a88d6eb[9](https://github.com/cilium/cilium/actions/runs/5185762997/jobs/9346045554?pr=25920#step:13:10)6bef83508805,clustermesh.apiserver.image.useDigest=false,debug.enabled=true,encryption.enabled=true,encryption.nodeEncryption=false,encryption.type=ipsec,extraConfig.monitor-aggregation=none,hubble.enabled=true,hubble.relay.enabled=true,hubble.relay.image.override=quay.io/cilium/hubble-relay-ci:6d5f174e837a7a4afbc3a88d6eb96bef83508805,hubble.relay.image.repository=quay.io/cilium/hubble-relay-ci,hubble.relay.image.tag=6d5f174e837a7a4afbc3a88d6eb96bef83508805,hubble.relay.image.useDigest=false,hubble.tls.ca.cert=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNFekNDQWJxZ0F3SUJBZ0lVUVVRTXpRRUV5VjNIRGpZd3JzT3UyVlBDMk1Rd0NnWUlLb1pJemowRUF3SXcKYURFTE1Ba0dBMVVFQmhNQ1ZWTXhGakFVQmdOVkJBZ1REVk5oYmlCR2NtRnVZMmx6WTI4eEN6QUpCZ05WQkFjVApBa05CTVE4d0RRWURWUVFLRXdaRGFXeHBkVzB4RHpBTkJnTlZCQXNUQmtOcGJHbDFiVEVTTUJBR0ExVUVBeE1KClEybHNhWFZ0SUVOQk1CNFhEVEl6TURZd05qQTNNekl3TUZvWERUSTRNRFl3TkRBM016SXdNRm93YURFTE1Ba0cKQTFVRUJoTUNWVk14RmpBVUJnTlZCQWdURFZOaGJpQkdjbUZ1WTJselkyOHhDekFKQmdOVkJBY1RBa05CTVE4dwpEUVlEVlFRS0V3WkRhV3hwZFcweER6QU5CZ05WQkFzVEJrTnBiR2wxYlRFU01CQUdBMVVFQXhNSlEybHNhWFZ0CklFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUUyd2ZiTHZLS045Z01lc2ZzTWxQUlM1OEgKRTI3YU1UL2owSEcwanlEdUtocnI0RHQ4cVdTK0l0eXJyR1VtS0xINllrb2FxdmxvS28ybGtIdHB1SDlWNXFOQwpNRUF3RGdZRFZSMFBBUUgvQkFRREFnRUdNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGQ1ppCkpXclFzN2tQWHUzeUxIQnBObWhia2liK01Bb0dDQ3FHU000OUJBTUNBMGNBTUVRQ0lGZElhUHNPSFUza2ZjWEQKUUNrNW9nRmhta2IxeUoybGJaS3ZkSjRoRHV1MUFpQUVUWktueVN2cnVsN003RGFTSkJydkhZc2xWZVdEL2ROTApKTm80TzE2N3BRPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=,hubble.tls.ca.key=[--- REDACTED WHEN PRINTING TO TERMINAL (USE --redact-helm-certificate-keys=false TO PRINT) ---],image.repository=quay.io/cilium/cilium-ci,image.tag=6d5f174e837a7a4afbc3a88d6eb96bef83508805,image.useDigest=false,ipam.mode=kubernetes,kubeProxyReplacement=disabled,operator.image.repository=quay.io/cilium/operator,operator.image.suffix=-ci,operator.image.tag=6d5f174e837a7a4afbc3a88d6eb96bef83508805,operator.image.useDigest=false,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
✨ Patching ConfigMap cilium-config to enable Hubble...
🚀 Creating ConfigMap for Cilium version 1.11.17...
♻️  Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
🚀 Creating Peer Service...
✨ Generating certificates...
🔑 Generating certificates for Relay...
✨ Deploying Relay...
⌛ Waiting for Hubble to be installed...

    /¯¯\
Error: Unable to enable Hubble: timeout while waiting for status to become successful: context deadline exceeded
 /¯¯\__/¯¯\    Cilium:         6 errors
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         1 errors, 1 warnings
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        hubble-relay       Desired: 1, Unavailable: 1/1
DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium-operator    Running: 1
                  cilium             Running: 2
                  hubble-relay       Pending: 1
Cluster Pods:     4/4 managed by Cilium
Image versions    hubble-relay       quay.io/cilium/hubble-relay-ci:6d5f174e837a7a4afbc3a88d6eb96bef83508805: 1
                  cilium-operator    quay.io/cilium/operator-generic-ci:6d5f174e837a7a4afbc3a88d6eb96bef83508805: 1
                  cilium             quay.io/cilium/cilium-ci:6d5f174e837a7a4afbc3a88d6eb96bef83508805: 2
Errors:           cilium             cilium-jzf7r                     controller cilium-health-ep is failing since 12s (11x): Get "http://[10](https://github.com/cilium/cilium/actions/runs/5185762997/jobs/9346045554?pr=25920#step:13:11).244.1.25:4240/hello": dial tcp 10.244.1.25:4240: connect: no route to host
                  cilium             cilium-jzf7r                     controller endpoint-7[11](https://github.com/cilium/cilium/actions/runs/5185762997/jobs/9346045554?pr=25920#step:13:12)-regeneration-recovery is failing since 23s (2x): regeneration recovery failed
                  cilium             cilium-s69ms                     controller endpoint-863-regeneration-recovery is failing since [13](https://github.com/cilium/cilium/actions/runs/5185762997/jobs/9346045554?pr=25920#step:13:14)s (5x): regeneration recovery failed
                  cilium             cilium-s69ms                     controller cilium-health-ep is failing since 39s (4x): Get "http://10.244.0.49:4240/hello": dial tcp 10.244.0.49:4240: connect: no route to host
                  cilium             cilium-s69ms                     controller endpoint-[14](https://github.com/cilium/cilium/actions/runs/5185762997/jobs/9346045554?pr=25920#step:13:15)[19](https://github.com/cilium/cilium/actions/runs/5185762997/jobs/9346045554?pr=25920#step:13:20)-regeneration-recovery is failing since 14s (5x): regeneration recovery failed
                  cilium             cilium-s69ms                     controller endpoint-495-regeneration-recovery is failing since 38s (4x): regeneration recovery failed
                  hubble-relay       hubble-relay                     1 pods of Deployment hubble-relay are not ready
Warnings:         hubble-relay       hubble-relay-5688fbb664-n4trh    pod is pending

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/CIContinuous Integration testing issue or flakeci/flakeThis is a known failure that occurs in the tree. Please investigate me!

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions