CI: Cilium K8s Client connection reset by peer #25958
Description
Test Name
K8sDatapathConfig Iptables Skip conntrack for pod traffic
Failure Output
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
Stacktrace
Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.26-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:415
Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-06-06T21:02:22.618917036Z level=error msg="Unexpected error when reading response body: read tcp 192.168.56.12:58686->192.168.56.11:6443: read: connection reset by peer" subsys=klog
2023-06-06T21:02:22.619048953Z level=error msg=k8sError error="github.com/cilium/cilium/pkg/k8s/resource/resource.go:253: Failed to watch *v2.CiliumIdentity: failed to list *v2.CiliumIdentity: unexpected error when reading response body. Please retry. Original error: read tcp 192.168.56.12:58686->192.168.56.11:6443: read: connection reset by peer" subsys=k8s
/home/jenkins/workspace/Cilium-PR-K8s-1.26-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:413
Standard Output
Click to show.
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
⚠️ Found "2023-06-06T21:02:22.618917036Z level=error msg=\"Unexpected error when reading response body: read tcp 192.168.56.12:58686->192.168.56.11:6443: read: connection reset by peer\" subsys=klog" in logs 1 times
⚠️ Found "2023-06-06T21:02:22.619048953Z level=error msg=k8sError error=\"github.com/cilium/cilium/pkg/k8s/resource/resource.go:253: Failed to watch *v2.CiliumIdentity: failed to list *v2.CiliumIdentity: unexpected error when reading response body. Please retry. Original error: read tcp 192.168.56.12:58686->192.168.56.11:6443: read: connection reset by peer\" subsys=k8s" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 3
⚠️ Number of "level=warning" in logs: 13
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
UpdateIdentities: Skipping Delete of a non-existing identity
github.com/cilium/cilium/pkg/k8s/resource/resource.go:253: Failed to watch *v2.CiliumIdentity: failed to list *v2.CiliumIdentity: unexpected error when reading response body. Please retry. Original error: read tcp 192.168.56.12:58686->192.168.56.11:6443: read: connection reset by peer
removing identity not added to the identity manager!
Unexpected error when reading response body: read tcp 192.168.56.12:58686->192.168.56.11:6443: read: connection reset by peer
github.com/cilium/cilium/pkg/k8s/resource/resource.go:253: watch of *v2.CiliumClusterwideNetworkPolicy ended with: very short watch: github.com/cilium/cilium/pkg/k8s/resource/resource.go:253: Unexpected watch close - watch lasted less than a second and no items received
Cilium pods: [cilium-fs5vv cilium-mw4cr]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
Pod Ingress Egress
testclient-g5cf9 false false
testds-2mbbx false false
testds-nqgrk false false
coredns-6d97d5ddb-8fkgg false false
test-k8s2-f5fdd6457-rhlcv false false
testclient-8v57x false false
Cilium agent 'cilium-fs5vv': Status: Ok Health: Ok Nodes "" ContainerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 30 Failed 0
Cilium agent 'cilium-mw4cr': Status: Ok Health: Ok Nodes "" ContainerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 38 Failed 0
Standard Error
Click to show.
21:02:10 STEP: Installing Cilium
21:02:12 STEP: Waiting for Cilium to become ready
21:02:24 STEP: Validating if Kubernetes DNS is deployed
21:02:24 STEP: Checking if deployment is ready
21:02:24 STEP: Checking if kube-dns service is plumbed correctly
21:02:24 STEP: Checking if pods have identity
21:02:24 STEP: Checking if DNS can resolve
21:02:28 STEP: Kubernetes DNS is not ready: %!s(<nil>)
21:02:28 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
21:02:28 STEP: Waiting for Kubernetes DNS to become operational
21:02:28 STEP: Checking if deployment is ready
21:02:29 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:02:29 STEP: Checking if deployment is ready
21:02:30 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:02:30 STEP: Checking if deployment is ready
21:02:31 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:02:31 STEP: Checking if deployment is ready
21:02:32 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:02:32 STEP: Checking if deployment is ready
21:02:33 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:02:33 STEP: Checking if deployment is ready
21:02:34 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:02:34 STEP: Checking if deployment is ready
21:02:35 STEP: Checking if kube-dns service is plumbed correctly
21:02:35 STEP: Checking if DNS can resolve
21:02:35 STEP: Checking if pods have identity
21:02:38 STEP: Validating Cilium Installation
21:02:38 STEP: Performing Cilium controllers preflight check
21:02:38 STEP: Performing Cilium status preflight check
21:02:38 STEP: Performing Cilium health check
21:02:38 STEP: Checking whether host EP regenerated
21:02:46 STEP: Performing Cilium service preflight check
21:02:46 STEP: Performing K8s service preflight check
21:02:52 STEP: Waiting for cilium-operator to be ready
21:02:52 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
21:02:52 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
21:02:52 STEP: Making sure all endpoints are in ready state
21:02:55 STEP: Creating namespace 202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic
21:02:55 STEP: Deploying demo_ds.yaml in namespace 202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic
21:02:56 STEP: Applying policy /home/jenkins/workspace/Cilium-PR-K8s-1.26-kernel-net-next/src/github.com/cilium/cilium/test/k8s/manifests/l3-policy-demo.yaml
21:03:04 STEP: Waiting for 4m0s for 5 pods of deployment demo_ds.yaml to become ready
21:03:04 STEP: WaitforNPods(namespace="202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic", filter="")
21:03:04 STEP: WaitforNPods(namespace="202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic", filter="") => <nil>
21:03:04 STEP: Checking pod connectivity between nodes
21:03:04 STEP: WaitforPods(namespace="202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic", filter="-l zgroup=testDSClient")
21:03:04 STEP: WaitforPods(namespace="202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic", filter="-l zgroup=testDSClient") => <nil>
21:03:04 STEP: WaitforPods(namespace="202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic", filter="-l zgroup=testDS")
21:03:04 STEP: WaitforPods(namespace="202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic", filter="-l zgroup=testDS") => <nil>
=== Test Finished at 2023-06-06T21:03:17Z====
21:03:17 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:
2023-06-06T21:02:22.618917036Z level=error msg="Unexpected error when reading response body: read tcp 192.168.56.12:58686->192.168.56.11:6443: read: connection reset by peer" subsys=klog
2023-06-06T21:02:22.619048953Z level=error msg=k8sError error="github.com/cilium/cilium/pkg/k8s/resource/resource.go:253: Failed to watch *v2.CiliumIdentity: failed to list *v2.CiliumIdentity: unexpected error when reading response body. Please retry. Original error: read tcp 192.168.56.12:58686->192.168.56.11:6443: read: connection reset by peer" subsys=k8s
===================== TEST FAILED =====================
21:03:18 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic test-k8s2-f5fdd6457-rhlcv 2/2 Running 0 27s 10.0.0.44 k8s2 <none> <none>
202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic testclient-8v57x 1/1 Running 0 27s 10.0.1.35 k8s1 <none> <none>
202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic testclient-g5cf9 1/1 Running 0 27s 10.0.0.97 k8s2 <none> <none>
202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic testds-2mbbx 2/2 Running 0 27s 10.0.0.90 k8s2 <none> <none>
202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic testds-nqgrk 2/2 Running 0 27s 10.0.1.101 k8s1 <none> <none>
cilium-monitoring grafana-67ff49cd99-rqhcm 0/1 Running 0 52m 10.0.0.104 k8s1 <none> <none>
cilium-monitoring prometheus-8c7df94b4-kncjr 1/1 Running 0 52m 10.0.0.241 k8s1 <none> <none>
kube-system cilium-fs5vv 1/1 Running 0 71s 192.168.56.11 k8s1 <none> <none>
kube-system cilium-mw4cr 1/1 Running 0 71s 192.168.56.12 k8s2 <none> <none>
kube-system cilium-operator-5d99db4564-bkzn6 1/1 Running 0 71s 192.168.56.13 k8s3 <none> <none>
kube-system cilium-operator-5d99db4564-h64xc 1/1 Running 0 71s 192.168.56.12 k8s2 <none> <none>
kube-system coredns-6d97d5ddb-8fkgg 1/1 Running 0 55s 10.0.0.173 k8s2 <none> <none>
kube-system etcd-k8s1 1/1 Running 0 59m 192.168.56.11 k8s1 <none> <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 59m 192.168.56.11 k8s1 <none> <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 59m 192.168.56.11 k8s1 <none> <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 59m 192.168.56.11 k8s1 <none> <none>
kube-system log-gatherer-444mm 1/1 Running 0 52m 192.168.56.11 k8s1 <none> <none>
kube-system log-gatherer-g8gd9 1/1 Running 0 52m 192.168.56.12 k8s2 <none> <none>
kube-system log-gatherer-ncrv7 1/1 Running 0 52m 192.168.56.13 k8s3 <none> <none>
kube-system registry-adder-khlbn 1/1 Running 0 53m 192.168.56.11 k8s1 <none> <none>
kube-system registry-adder-mw85q 1/1 Running 0 53m 192.168.56.12 k8s2 <none> <none>
kube-system registry-adder-pl2bx 1/1 Running 0 53m 192.168.56.13 k8s3 <none> <none>
Stderr:
Fetching command output from pods [cilium-fs5vv cilium-mw4cr]
cmd: kubectl exec -n kube-system cilium-fs5vv -c cilium-agent -- cilium status
Exitcode: 0
Stdout:
KVStore: Ok Disabled
Kubernetes: Ok 1.26 (v1.26.3) [linux/amd64]
Kubernetes APIs: ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fec4:5914, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
Host firewall: Disabled
CNI Chaining: none
Cilium: Ok 1.14.0-dev (v1.14.0-dev-2a4924f7)
NodeMonitor: Listening for events on 3 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 4/254 allocated from 10.0.1.0/24, IPv6: 4/254 allocated from fd02::100/120
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Host Routing: BPF
Masquerading: BPF [enp0s16, enp0s3, enp0s8] 10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
Controller Status: 30/30 healthy
Proxy Status: OK, ip 10.0.1.173, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range: min 256, max 65535
Hubble: Ok Current/Max Flows: 828/65535 (1.26%), Flows/s: 6.17 Metrics: Disabled
Encryption: Disabled
Cluster health: 2/2 reachable (2023-06-06T21:02:45Z)
Stderr:
cmd: kubectl exec -n kube-system cilium-fs5vv -c cilium-agent -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
397 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s1 ready
k8s:node-role.kubernetes.io/control-plane
k8s:node.kubernetes.io/exclude-from-external-load-balancers
reserved:host
542 Disabled Disabled 4 reserved:health fd02::1ba 10.0.1.180 ready
1381 Disabled Disabled 32412 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic fd02::17c 10.0.1.101 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic
k8s:zgroup=testDS
3969 Disabled Disabled 29421 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic fd02::149 10.0.1.35 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic
k8s:zgroup=testDSClient
Stderr:
cmd: kubectl exec -n kube-system cilium-mw4cr -c cilium-agent -- cilium status
Exitcode: 0
Stdout:
KVStore: Ok Disabled
Kubernetes: Ok 1.26 (v1.26.3) [linux/amd64]
Kubernetes APIs: ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fee2:a8b5, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
Host firewall: Disabled
CNI Chaining: none
Cilium: Ok 1.14.0-dev (v1.14.0-dev-2a4924f7)
NodeMonitor: Listening for events on 3 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 6/254 allocated from 10.0.0.0/24, IPv6: 6/254 allocated from fd02::/120
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Host Routing: BPF
Masquerading: BPF [enp0s16, enp0s3, enp0s8] 10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
Controller Status: 38/38 healthy
Proxy Status: OK, ip 10.0.0.107, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range: min 256, max 65535
Hubble: Ok Current/Max Flows: 1090/65535 (1.66%), Flows/s: 10.17 Metrics: Disabled
Encryption: Disabled
Cluster health: 2/2 reachable (2023-06-06T21:02:52Z)
Stderr:
cmd: kubectl exec -n kube-system cilium-mw4cr -c cilium-agent -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
116 Disabled Disabled 33352 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic fd02::cb 10.0.0.44 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic
k8s:zgroup=test-k8s2
132 Disabled Disabled 32412 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic fd02::9c 10.0.0.90 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic
k8s:zgroup=testDS
1408 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s2 ready
reserved:host
1440 Disabled Disabled 29421 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic fd02::f1 10.0.0.97 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic
k8s:zgroup=testDSClient
1835 Disabled Disabled 4 reserved:health fd02::65 10.0.0.103 ready
2020 Disabled Disabled 9444 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system fd02::66 10.0.0.173 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
Stderr:
===================== Exiting AfterFailed =====================
21:03:51 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
21:03:51 STEP: Deleting deployment demo_ds.yaml
21:03:52 STEP: Deleting namespace 202306062102k8sdatapathconfigiptablesskipconntrackforpodtraffic
21:04:05 STEP: Running AfterEach for block EntireTestsuite
[[ATTACHMENT|431a669a_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
21:04:06 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
21:04:13 STEP: Deleting Cilium
ZIP Links:
Click to show.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next//521/artifact/431a669a_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next//521/artifact/cilium-sysdump.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next//521/artifact/test_results_Cilium-PR-K8s-1.26-kernel-net-next_521_BDD-Test-PR.zip
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next/521/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.