CI: K8sUpdates: Missed tail calls #24514
Description
Test Name
K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master
Failure Output
cmd: "kubectl exec -n kube-system cilium-dj8v9 -- cilium metrics list -o json | jq '.[] | select( .name == \"cilium_drop_count_total\" and .labels.reason == \"Missed tail call\" ).value'" exitCode: 0 duration: 168.910676ms stdout:
1
FAIL: Expected
<int>: 1
to be ==
<int>: 0
Standard Output
Click to show.
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 2 errors/warnings:
UpdateIdentities: Skipping Delete of a non-existing identity
Attempt to remove non-existing IP from ipcache layer
Cilium pods: [cilium-csq2z cilium-dj8v9]
Netpols loaded:
CiliumNetworkPolicies loaded: default::l7-policy
Endpoint Policy Enforcement:
Pod Ingress Egress
app1-586cfd8997-8xz78 false false
app2-775964bd4-sqjm9 false false
migrate-svc-client-5lfg2 false false
migrate-svc-client-dxqbs false false
migrate-svc-server-w8rvb false false
coredns-6d97d5ddb-xcxjx false false
migrate-svc-server-z7ckd false false
app1-586cfd8997-t8zcj false false
app3-5db68b966f-9wxr5 false false
migrate-svc-client-ht4lk false false
migrate-svc-client-lvfjp false false
migrate-svc-client-qnwl2 false false
migrate-svc-server-w2lpb false false
Cilium agent 'cilium-csq2z': Status: Ok Health: Ok Nodes "" ContainerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 60 Failed 0
Cilium agent 'cilium-dj8v9': Status: Ok Health: Ok Nodes "" ContainerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 45 Failed 0
Standard Error
Click to show.
21:12:30 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
21:12:30 STEP: Ensuring the namespace kube-system exists
21:12:30 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
21:12:30 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
21:12:30 STEP: Waiting for pods to be terminated
21:12:37 STEP: Deleting Cilium and CoreDNS
21:12:37 STEP: Waiting for pods to be terminated
21:12:37 STEP: Cleaning Cilium state (7758b3ee9def91a3c53cb6c6a1161d203a04b9f2)
21:12:37 STEP: Cleaning up Cilium components
21:12:39 STEP: Waiting for Cilium to become ready
21:12:52 STEP: Cleaning Cilium state (v1.13)
21:12:52 STEP: Cleaning up Cilium components
21:12:53 STEP: Waiting for Cilium to become ready
21:13:36 STEP: Deploying Cilium 1.13
21:13:37 STEP: Waiting for Cilium to become ready
21:14:28 STEP: Validating Cilium Installation
21:14:28 STEP: Performing Cilium controllers preflight check
21:14:28 STEP: Performing Cilium status preflight check
21:14:28 STEP: Checking whether host EP regenerated
21:14:28 STEP: Performing Cilium health check
21:14:47 STEP: Performing Cilium service preflight check
21:14:47 STEP: Cilium is not ready yet: host EP is not ready: cilium-agent "cilium-b22np" host EP is not in ready state: "waiting-to-regenerate"
21:14:47 STEP: Performing Cilium controllers preflight check
21:14:47 STEP: Performing Cilium status preflight check
21:14:47 STEP: Performing Cilium health check
21:14:47 STEP: Checking whether host EP regenerated
21:15:05 STEP: Performing Cilium service preflight check
21:15:05 STEP: Performing K8s service preflight check
21:15:05 STEP: Waiting for cilium-operator to be ready
21:15:05 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
21:15:05 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
21:15:05 STEP: Cilium "1.13" is installed and running
21:15:05 STEP: Restarting DNS Pods
21:15:11 STEP: Waiting for kube-dns to be ready
21:15:11 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns")
21:15:11 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns") => <nil>
21:15:11 STEP: Running kube-dns preflight check
21:15:28 STEP: Performing K8s service preflight check
21:15:28 STEP: Creating some endpoints and L7 policy
21:15:29 STEP: WaitforPods(namespace="default", filter="-l zgroup=testapp")
21:15:33 STEP: WaitforPods(namespace="default", filter="-l zgroup=testapp") => <nil>
21:15:41 STEP: Creating service and clients for migration
21:15:41 STEP: WaitforPods(namespace="default", filter="-l app=migrate-svc-server")
21:15:45 STEP: WaitforPods(namespace="default", filter="-l app=migrate-svc-server") => <nil>
21:15:45 STEP: WaitforPods(namespace="default", filter="-l app=migrate-svc-client")
21:15:48 STEP: WaitforPods(namespace="default", filter="-l app=migrate-svc-client") => <nil>
21:15:48 STEP: Validate that endpoints are ready before making any connection
21:15:50 STEP: Waiting for kube-dns to be ready
21:15:50 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns")
21:15:50 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns") => <nil>
21:15:50 STEP: Running kube-dns preflight check
21:16:07 STEP: Performing K8s service preflight check
21:16:08 STEP: Making L7 requests between endpoints
21:16:08 STEP: No interrupts in migrated svc flows
21:16:08 STEP: Install Cilium pre-flight check DaemonSet
21:16:09 STEP: Waiting for all cilium pre-flight pods to be ready
21:16:09 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-pre-flight-check")
21:16:20 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-pre-flight-check") => <nil>
21:16:20 STEP: Removing Cilium pre-flight check DaemonSet
21:16:21 STEP: Waiting for Cilium to become ready
21:16:21 STEP: Upgrading Cilium to 1.13.90
21:16:22 STEP: Validating pods have the right image version upgraded
21:16:24 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium")
21:16:51 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium") => <nil>
21:16:51 STEP: Checking that installed image is "7758b3ee9def91a3c53cb6c6a1161d203a04b9f2"
21:16:51 STEP: Waiting for Cilium to become ready
21:16:51 STEP: Validating Cilium Installation
21:16:51 STEP: Performing Cilium controllers preflight check
21:16:51 STEP: Performing Cilium status preflight check
21:16:51 STEP: Performing Cilium health check
21:16:51 STEP: Checking whether host EP regenerated
21:17:01 STEP: Performing Cilium service preflight check
21:17:01 STEP: Performing K8s service preflight check
21:17:01 STEP: Cilium is not ready yet: host EP is not ready: cilium-agent "cilium-csq2z" host EP is not in ready state: "regenerating"
21:17:01 STEP: Performing Cilium controllers preflight check
21:17:01 STEP: Performing Cilium health check
21:17:01 STEP: Checking whether host EP regenerated
21:17:01 STEP: Performing Cilium status preflight check
21:17:08 STEP: Performing Cilium service preflight check
21:17:08 STEP: Performing K8s service preflight check
21:17:08 STEP: Cilium is not ready yet: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-csq2z': Exitcode: 1
Err: exit status 1
Stdout:
Stderr:
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
Error: Cannot get status/probe: Put "http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe": dial unix /var/run/cilium/health.sock: connect: no such file or directory
command terminated with exit code 1
21:17:08 STEP: Performing Cilium controllers preflight check
21:17:08 STEP: Performing Cilium status preflight check
21:17:08 STEP: Performing Cilium health check
21:17:08 STEP: Checking whether host EP regenerated
21:17:16 STEP: Performing Cilium service preflight check
21:17:16 STEP: Performing K8s service preflight check
21:17:22 STEP: Waiting for cilium-operator to be ready
21:17:22 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
21:17:22 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
21:17:22 STEP: Validate that endpoints are ready before making any connection
21:17:25 STEP: Waiting for kube-dns to be ready
21:17:25 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns")
21:17:25 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=kube-dns") => <nil>
21:17:25 STEP: Running kube-dns preflight check
21:17:32 STEP: Performing K8s service preflight check
21:17:33 STEP: Making L7 requests between endpoints
21:17:33 STEP: No interrupts in migrated svc flows
FAIL: Expected
<int>: 1
to be ==
<int>: 0
=== Test Finished at 2023-03-21T21:17:37Z====
21:17:37 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
21:17:37 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cilium-monitoring grafana-67ff49cd99-9mbh7 0/1 Running 0 16m 10.0.0.246 k8s1 <none> <none>
cilium-monitoring prometheus-8c7df94b4-gxqdz 1/1 Running 0 16m 10.0.0.66 k8s1 <none> <none>
default app1-586cfd8997-8xz78 2/2 Running 0 2m13s 10.0.1.139 k8s1 <none> <none>
default app1-586cfd8997-t8zcj 2/2 Running 0 2m13s 10.0.1.160 k8s1 <none> <none>
default app2-775964bd4-sqjm9 1/1 Running 0 2m13s 10.0.1.227 k8s1 <none> <none>
default app3-5db68b966f-9wxr5 1/1 Running 0 2m13s 10.0.1.213 k8s1 <none> <none>
default migrate-svc-client-5lfg2 1/1 Running 0 117s 10.0.0.9 k8s2 <none> <none>
default migrate-svc-client-dxqbs 1/1 Running 0 117s 10.0.0.196 k8s2 <none> <none>
default migrate-svc-client-ht4lk 1/1 Running 0 117s 10.0.1.58 k8s1 <none> <none>
default migrate-svc-client-lvfjp 1/1 Running 0 117s 10.0.1.123 k8s1 <none> <none>
default migrate-svc-client-qnwl2 1/1 Running 0 117s 10.0.0.150 k8s2 <none> <none>
default migrate-svc-server-w2lpb 1/1 Running 0 2m1s 10.0.1.192 k8s1 <none> <none>
default migrate-svc-server-w8rvb 1/1 Running 0 2m1s 10.0.0.25 k8s2 <none> <none>
default migrate-svc-server-z7ckd 1/1 Running 0 2m1s 10.0.0.118 k8s2 <none> <none>
kube-system cilium-csq2z 1/1 Running 0 79s 192.168.56.11 k8s1 <none> <none>
kube-system cilium-dj8v9 1/1 Running 0 79s 192.168.56.12 k8s2 <none> <none>
kube-system cilium-operator-6557cc55f-42h69 1/1 Running 0 80s 192.168.56.13 k8s3 <none> <none>
kube-system cilium-operator-6557cc55f-lhzwq 1/1 Running 0 80s 192.168.56.11 k8s1 <none> <none>
kube-system coredns-6d97d5ddb-xcxjx 1/1 Running 0 2m37s 10.0.1.87 k8s1 <none> <none>
kube-system etcd-k8s1 1/1 Running 0 26m 192.168.56.11 k8s1 <none> <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 26m 192.168.56.11 k8s1 <none> <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 26m 192.168.56.11 k8s1 <none> <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 26m 192.168.56.11 k8s1 <none> <none>
kube-system log-gatherer-bvrc9 1/1 Running 0 16m 192.168.56.13 k8s3 <none> <none>
kube-system log-gatherer-kzbws 1/1 Running 0 16m 192.168.56.12 k8s2 <none> <none>
kube-system log-gatherer-wzk4z 1/1 Running 0 16m 192.168.56.11 k8s1 <none> <none>
kube-system registry-adder-c2wzm 1/1 Running 0 17m 192.168.56.11 k8s1 <none> <none>
kube-system registry-adder-n85bq 1/1 Running 0 17m 192.168.56.13 k8s3 <none> <none>
kube-system registry-adder-wdtcs 1/1 Running 0 17m 192.168.56.12 k8s2 <none> <none>
Stderr:
Fetching command output from pods [cilium-csq2z cilium-dj8v9]
cmd: kubectl exec -n kube-system cilium-csq2z -c cilium-agent -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
279 Disabled Disabled 55951 k8s:app=migrate-svc-client fd02::130 10.0.1.123 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=migrate-svc
543 Disabled Disabled 54668 k8s:appSecond=true fd02::1f4 10.0.1.227 ready
k8s:id=app2
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=app2-account
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testapp
566 Disabled Disabled 3983 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system fd02::1df 10.0.1.87 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
921 Disabled Disabled 38367 k8s:app=migrate-svc-server fd02::1f1 10.0.1.192 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=migrate-svc
942 Enabled Disabled 857 k8s:id=app1 fd02::114 10.0.1.160 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=app1-account
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testapp
1020 Disabled Disabled 35668 k8s:id=app3 fd02::13d 10.0.1.213 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testapp
1173 Disabled Disabled 4 reserved:health fd02::104 10.0.1.40 ready
1685 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s1 ready
k8s:node-role.kubernetes.io/control-plane
k8s:node.kubernetes.io/exclude-from-external-load-balancers
reserved:host
3353 Enabled Disabled 857 k8s:id=app1 fd02::1a9 10.0.1.139 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=app1-account
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testapp
3484 Disabled Disabled 55951 k8s:app=migrate-svc-client fd02::10d 10.0.1.58 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=migrate-svc
Stderr:
cmd: kubectl exec -n kube-system cilium-dj8v9 -c cilium-agent -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
1197 Disabled Disabled 55951 k8s:app=migrate-svc-client fd02::2c 10.0.0.196 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=migrate-svc
1410 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s2 ready
reserved:host
2531 Disabled Disabled 55951 k8s:app=migrate-svc-client fd02::2b 10.0.0.9 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=migrate-svc
2626 Disabled Disabled 4 reserved:health fd02::be 10.0.0.17 ready
3317 Disabled Disabled 38367 k8s:app=migrate-svc-server fd02::9 10.0.0.118 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=migrate-svc
3731 Disabled Disabled 55951 k8s:app=migrate-svc-client fd02::30 10.0.0.150 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=migrate-svc
4028 Disabled Disabled 38367 k8s:app=migrate-svc-server fd02::ad 10.0.0.25 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=migrate-svc
Stderr:
===================== Exiting AfterFailed =====================
21:17:51 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
21:17:57 STEP: Cleaning up Cilium components
21:18:00 STEP: Waiting for Cilium to become ready
21:18:14 STEP: Running AfterEach for block EntireTestsuite
[[ATTACHMENT|d93248ca_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
21:18:16 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
21:18:16 STEP: Cleaning up Cilium components
ZIP Links:
Click to show.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next//1412/artifact/d93248ca_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next//1412/artifact/test_results_Cilium-PR-K8s-1.26-kernel-net-next_1412_BDD-Test-PR.zip
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next/1412/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.