CI: External workloads - connectivity test failed #25636
Description
CI failure
Run: https://github.com/cilium/cilium/actions/runs/5055963581/jobs/9072859869
Sysdump: cilium-sysdump-final.zip
External workload log: external-workloads.log
Connectivity test:
Tue, 23 May 2023 10:19:04 GMT ✨ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Deploying echo-same-node service...
Tue, 23 May 2023 10:19:04 GMT ✨ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Deploying DNS test server configmap...
Tue, 23 May 2023 10:19:04 GMT ✨ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Deploying same-node deployment...
Tue, 23 May 2023 10:19:04 GMT ✨ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Deploying client deployment...
Tue, 23 May 2023 10:19:04 GMT ✨ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Deploying client2 deployment...
Tue, 23 May 2023 10:19:05 GMT ✨ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Deploying echo-other-node service...
Tue, 23 May 2023 10:19:05 GMT ✨ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Deploying other-node deployment...
Tue, 23 May 2023 10:19:05 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for deployments [client client2 echo-same-node] to become ready...
Tue, 23 May 2023 10:22:16 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for deployments [echo-other-node] to become ready...
Tue, 23 May 2023 10:22:16 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for CiliumEndpoint for pod cilium-test/client-7b78db77d5-x27zv to appear...
Tue, 23 May 2023 10:22:16 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for CiliumEndpoint for pod cilium-test/client2-78f748dd67-tmfxs to appear...
Tue, 23 May 2023 10:22:17 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for pod cilium-test/client-7b78db77d5-x27zv to reach DNS server on cilium-test/echo-same-node-5b55d8bdd7-fqk9r pod...
Tue, 23 May 2023 10:22:17 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for pod cilium-test/client2-78f748dd67-tmfxs to reach DNS server on cilium-test/echo-same-node-5b55d8bdd7-fqk9r pod...
Tue, 23 May 2023 10:22:18 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for pod cilium-test/client-7b78db77d5-x27zv to reach DNS server on cilium-test/echo-other-node-8689b5dd4f-4hpdq pod...
Tue, 23 May 2023 10:22:18 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for pod cilium-test/client2-78f748dd67-tmfxs to reach DNS server on cilium-test/echo-other-node-8689b5dd4f-4hpdq pod...
Tue, 23 May 2023 10:22:19 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for pod cilium-test/client2-78f748dd67-tmfxs to reach default/kubernetes service...
Tue, 23 May 2023 10:22:19 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for pod cilium-test/client-7b78db77d5-x27zv to reach default/kubernetes service...
Tue, 23 May 2023 10:22:20 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for CiliumEndpoint for pod cilium-test/echo-other-node-8689b5dd4f-4hpdq to appear...
Tue, 23 May 2023 10:22:20 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for CiliumEndpoint for pod cilium-test/echo-same-node-5b55d8bdd7-fqk9r to appear...
Tue, 23 May 2023 10:22:20 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for Service cilium-test/echo-other-node to become ready...
Tue, 23 May 2023 10:22:20 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for Service cilium-test/echo-same-node to become ready...
Tue, 23 May 2023 10:22:21 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for NodePort 10.168.15.203:30271 (cilium-test/echo-same-node) to become ready...
Tue, 23 May 2023 10:22:21 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for NodePort 10.168.15.203:32264 (cilium-test/echo-other-node) to become ready...
Tue, 23 May 2023 10:22:22 GMT ⌛ [gke_***_us-west2-a_cilium-cilium-5055963581-vm] Waiting for NodePort 10.168.0.15:32264 (cilium-test/echo-other-node) to become ready...
Tue, 23 May 2023 10:22:52 GMT connectivity test failed: timeout reached waiting for NodePort 10.168.0.15:32264 (cilium-test/echo-other-node) (last error: context deadline exceeded)
Status of cilium running in the VM:
⚠️ Cluster not configured for clustermesh, use '--cluster-id' and '--cluster-name' with 'cilium install'. External workloads may still be configured.
✅ Cluster access information is available:
- 10.168.15.209:2379
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
🔌 Cluster Connections:
🔀 Global services: [ min:-1 / avg:0.0 / max:0 ]
✅ Cluster access information is available:
- 10.168.15.209:2379
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
External Workloads cilium-cilium-5055963581-vm IP: 10.168.0.127, Labels:
KVStore: Failure Err: quorum check failed 3 times in a row: 3m26.746343853s since last heartbeat update has been received
Kubernetes: Disabled
Host firewall: Disabled
CNI Chaining: none
Cilium: Failure 1.14.0-dev (v1.14.0-dev-0f3f4f94) Kvstore service is not ready
NodeMonitor: Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 1/2 allocated from 10.192.1.0/30, IPv6: 1/4294967294 allocated from f00d::a7f:0:0:0/96
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: IPTables [IPv4: Enabled, IPv6: Enabled]
Controller Status: 17/17 healthy
Proxy Status: OK, ip 10.192.1.2, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range: min 256, max 65535
Hubble: Disabled
Encryption: Disabled