CI: K8sDatapathConfig AutoDirectNodeRoutes Check connectivity with sockops and direct routing #17373
Description
Test Name
K8sDatapathConfig AutoDirectNodeRoutes Check connectivity with sockops and direct routing
Failure Output
FAIL: Error creating resource /home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-5.4/src/github.com/cilium/cilium/test/k8sT/manifests/l3-policy-demo.yaml: Cannot retrieve cilium pod cilium-nctq6 policy revision: cannot get revision from json output '': could not parse JSON from command "kubectl exec -n kube-system cilium-nctq6 -- cilium policy get -o json"
Stacktrace
Click to show.
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Error creating resource /home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-5.4/src/github.com/cilium/cilium/test/k8sT/manifests/l3-policy-demo.yaml: Cannot retrieve cilium pod cilium-nctq6 policy revision: cannot get revision from json output '': could not parse JSON from command "kubectl exec -n kube-system cilium-nctq6 -- cilium policy get -o json"
unexpected end of JSON input
Expected
<*errors.errorString | 0xc001154490>: {
s: "Cannot retrieve cilium pod cilium-nctq6 policy revision: cannot get revision from json output '': could not parse JSON from command \"kubectl exec -n kube-system cilium-nctq6 -- cilium policy get -o json\"\nunexpected end of JSON input\n",
}
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-5.4/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:1039
Standard Output
Click to show.
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 2
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-nctq6 cilium-nt8g5]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
Pod Ingress Egress
testds-g4qm2
testds-zbqln
grafana-d69c97b9b-cvjhz
prometheus-655fb888d7-5zq5j
coredns-867bf6789f-mlsh6
test-k8s2-79ff876c9d-j6bf7
testclient-nwtsd
testclient-xsgqw
Cilium agent 'cilium-nctq6': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 36 Failed 0
Cilium agent 'cilium-nt8g5': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 34 Failed 0
Standard Error
Click to show.
21:55:29 STEP: Running BeforeEach block for EntireTestsuite K8sDatapathConfig AutoDirectNodeRoutes
21:55:29 STEP: Installing Cilium
21:55:31 STEP: Waiting for Cilium to become ready
21:56:19 STEP: Validating if Kubernetes DNS is deployed
21:56:19 STEP: Checking if deployment is ready
21:56:19 STEP: Checking if kube-dns service is plumbed correctly
21:56:19 STEP: Checking if pods have identity
21:56:19 STEP: Checking if DNS can resolve
21:56:20 STEP: Kubernetes DNS is not ready: %!s(<nil>)
21:56:20 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
21:56:27 STEP: Waiting for Kubernetes DNS to become operational
21:56:27 STEP: Checking if deployment is ready
21:56:27 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:56:28 STEP: Checking if deployment is ready
21:56:28 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:56:29 STEP: Checking if deployment is ready
21:56:29 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:56:30 STEP: Checking if deployment is ready
21:56:30 STEP: Checking if kube-dns service is plumbed correctly
21:56:30 STEP: Checking if pods have identity
21:56:30 STEP: Checking if DNS can resolve
21:56:31 STEP: Validating Cilium Installation
21:56:31 STEP: Performing Cilium health check
21:56:31 STEP: Performing Cilium status preflight check
21:56:31 STEP: Performing Cilium controllers preflight check
21:56:32 STEP: Performing Cilium service preflight check
21:56:32 STEP: Performing K8s service preflight check
21:56:32 STEP: Cilium is not ready yet: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-nctq6': Exitcode: 1
Err: exit status 1
Stdout:
Stderr:
Error: Cannot get status/probe: Put "http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe": dial unix /var/run/cilium/health.sock: connect: no such file or directory
command terminated with exit code 1
21:56:32 STEP: Performing Cilium controllers preflight check
21:56:32 STEP: Performing Cilium status preflight check
21:56:32 STEP: Performing Cilium health check
21:56:35 STEP: Performing Cilium service preflight check
21:56:35 STEP: Performing K8s service preflight check
21:56:35 STEP: Cilium is not ready yet: connectivity health is failing: Cannot unmarshal health status: could not parse JSON from command "kubectl exec -n kube-system cilium-nctq6 -- cilium-health status -o json --probe"
unexpected end of JSON input
21:56:35 STEP: Performing Cilium controllers preflight check
21:56:35 STEP: Performing Cilium health check
21:56:35 STEP: Performing Cilium status preflight check
21:56:36 STEP: Performing Cilium service preflight check
21:56:36 STEP: Performing K8s service preflight check
21:56:36 STEP: Cilium is not ready yet: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-nctq6': Exitcode: 1
Err: exit status 1
Stdout:
Stderr:
Error: Cannot get status/probe: Put "http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe": dial unix /var/run/cilium/health.sock: connect: no such file or directory
command terminated with exit code 1
21:56:36 STEP: Performing Cilium controllers preflight check
21:56:36 STEP: Performing Cilium health check
21:56:36 STEP: Performing Cilium status preflight check
21:56:38 STEP: Performing Cilium service preflight check
21:56:38 STEP: Performing K8s service preflight check
21:56:38 STEP: Cilium is not ready yet: connectivity health is failing: Cannot unmarshal health status: could not parse JSON from command "kubectl exec -n kube-system cilium-nctq6 -- cilium-health status -o json --probe"
unexpected end of JSON input
21:56:38 STEP: Performing Cilium status preflight check
21:56:38 STEP: Performing Cilium health check
21:56:38 STEP: Performing Cilium controllers preflight check
21:56:42 STEP: Performing Cilium service preflight check
21:56:42 STEP: Performing K8s service preflight check
21:56:42 STEP: Waiting for cilium-operator to be ready
21:56:43 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
21:56:43 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
21:56:43 STEP: Making sure all endpoints are in ready state
21:56:44 STEP: Creating namespace 202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv
21:56:44 STEP: Deploying demo_ds.yaml in namespace 202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv
21:56:45 STEP: Applying policy /home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-5.4/src/github.com/cilium/cilium/test/k8sT/manifests/l3-policy-demo.yaml
FAIL: Error creating resource /home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-5.4/src/github.com/cilium/cilium/test/k8sT/manifests/l3-policy-demo.yaml: Cannot retrieve cilium pod cilium-nctq6 policy revision: cannot get revision from json output '': could not parse JSON from command "kubectl exec -n kube-system cilium-nctq6 -- cilium policy get -o json"
unexpected end of JSON input
Expected
<*errors.errorString | 0xc001154490>: {
s: "Cannot retrieve cilium pod cilium-nctq6 policy revision: cannot get revision from json output '': could not parse JSON from command \"kubectl exec -n kube-system cilium-nctq6 -- cilium policy get -o json\"\nunexpected end of JSON input\n",
}
to be nil
=== Test Finished at 2021-09-10T21:56:45Z====
21:56:45 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
21:56:46 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv test-k8s2-79ff876c9d-j6bf7 1/2 Running 0 5s 10.0.1.242 k8s2 <none> <none>
202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv testclient-nwtsd 0/1 ContainerCreating 0 6s <none> k8s1 <none> <none>
202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv testclient-xsgqw 1/1 Running 0 6s 10.0.1.11 k8s2 <none> <none>
202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv testds-g4qm2 0/2 ContainerCreating 0 6s <none> k8s1 <none> <none>
202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv testds-zbqln 1/2 Running 0 6s 10.0.1.245 k8s2 <none> <none>
cilium-monitoring grafana-d69c97b9b-cvjhz 1/1 Running 0 18m 10.0.0.227 k8s1 <none> <none>
cilium-monitoring prometheus-655fb888d7-5zq5j 1/1 Running 0 18m 10.0.0.78 k8s1 <none> <none>
kube-system cilium-nctq6 1/1 Running 0 79s 192.168.36.11 k8s1 <none> <none>
kube-system cilium-nt8g5 1/1 Running 0 79s 192.168.36.12 k8s2 <none> <none>
kube-system cilium-operator-57fb97c55b-gx2cz 1/1 Running 0 79s 192.168.36.11 k8s1 <none> <none>
kube-system cilium-operator-57fb97c55b-srpgb 1/1 Running 0 79s 192.168.36.12 k8s2 <none> <none>
kube-system coredns-867bf6789f-mlsh6 1/1 Running 0 30s 10.0.1.155 k8s2 <none> <none>
kube-system etcd-k8s1 1/1 Running 0 21m 192.168.36.11 k8s1 <none> <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 21m 192.168.36.11 k8s1 <none> <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 21m 192.168.36.11 k8s1 <none> <none>
kube-system kube-proxy-s4mm4 1/1 Running 0 19m 192.168.36.12 k8s2 <none> <none>
kube-system kube-proxy-w7mtc 1/1 Running 0 20m 192.168.36.11 k8s1 <none> <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 21m 192.168.36.11 k8s1 <none> <none>
kube-system log-gatherer-jw65v 1/1 Running 0 18m 192.168.36.11 k8s1 <none> <none>
kube-system log-gatherer-tntcc 1/1 Running 0 18m 192.168.36.12 k8s2 <none> <none>
kube-system registry-adder-lf7kp 1/1 Running 0 18m 192.168.36.11 k8s1 <none> <none>
kube-system registry-adder-w98nc 1/1 Running 0 18m 192.168.36.12 k8s2 <none> <none>
Stderr:
Fetching command output from pods [cilium-nctq6 cilium-nt8g5]
cmd: kubectl exec -n kube-system cilium-nctq6 -c cilium-agent -- cilium status
Exitcode: 0
Stdout:
KVStore: Ok Disabled
Kubernetes: Ok 1.19 (v1.19.13) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.36.11 fd04::11 (Direct Routing)]
Host firewall: Disabled
Cilium: Ok 1.10.90 (v1.10.90-5d1af6a)
NodeMonitor: Listening for events on 3 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 6/254 allocated from 10.0.0.0/24, IPv6: 6/254 allocated from fd02::/120
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: BPF [enp0s3, enp0s8] 10.0.0.0/8 [IPv4: Enabled, IPv6: Enabled]
Controller Status: 36/36 healthy
Proxy Status: OK, ip 10.0.0.119, 0 redirects active on ports 10000-20000
Hubble: Ok Current/Max Flows: 332/4095 (8.11%), Flows/s: 5.01 Metrics: Disabled
Encryption: Disabled
Cluster health: 2/2 reachable (2021-09-10T21:56:40Z)
Stderr:
cmd: kubectl exec -n kube-system cilium-nctq6 -c cilium-agent -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
1 Disabled Disabled 4 reserved:health fd02::7a 10.0.0.156 ready
1402 Disabled Disabled 57145 k8s:app=grafana fd02::40 10.0.0.227 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=cilium-monitoring
1920 Disabled Disabled 29315 k8s:io.cilium.k8s.policy.cluster=default fd02::2f 10.0.0.40 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv
k8s:zgroup=testDSClient
1943 Disabled Disabled 22501 k8s:app=prometheus fd02::48 10.0.0.78 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s
k8s:io.kubernetes.pod.namespace=cilium-monitoring
2508 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s1 ready
k8s:node-role.kubernetes.io/master
reserved:host
2708 Disabled Disabled 13914 k8s:io.cilium.k8s.policy.cluster=default fd02::4f 10.0.0.126 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv
k8s:zgroup=testDS
Stderr:
cmd: kubectl exec -n kube-system cilium-nt8g5 -c cilium-agent -- cilium status
Exitcode: 0
Stdout:
KVStore: Ok Disabled
Kubernetes: Ok 1.19 (v1.19.13) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.36.12 fd04::12 (Direct Routing)]
Host firewall: Disabled
Cilium: Ok 1.10.90 (v1.10.90-5d1af6a)
NodeMonitor: Listening for events on 3 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 6/254 allocated from 10.0.1.0/24, IPv6: 6/254 allocated from fd02::100/120
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: BPF [enp0s3, enp0s8] 10.0.0.0/8 [IPv4: Enabled, IPv6: Enabled]
Controller Status: 34/34 healthy
Proxy Status: OK, ip 10.0.1.85, 0 redirects active on ports 10000-20000
Hubble: Ok Current/Max Flows: 260/4095 (6.35%), Flows/s: 3.48 Metrics: Disabled
Encryption: Disabled
Cluster health: 2/2 reachable (2021-09-10T21:56:43Z)
Stderr:
cmd: kubectl exec -n kube-system cilium-nt8g5 -c cilium-agent -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
1438 Disabled Disabled 13914 k8s:io.cilium.k8s.policy.cluster=default fd02::194 10.0.1.245 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv
k8s:zgroup=testDS
1899 Disabled Disabled 47291 k8s:io.cilium.k8s.policy.cluster=default fd02::1bb 10.0.1.155 ready
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
2840 Disabled Disabled 9341 k8s:io.cilium.k8s.policy.cluster=default fd02::13d 10.0.1.242 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv
k8s:zgroup=test-k8s2
3722 Disabled Disabled 29315 k8s:io.cilium.k8s.policy.cluster=default fd02::1c0 10.0.1.11 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv
k8s:zgroup=testDSClient
3748 Disabled Disabled 4 reserved:health fd02::1b3 10.0.1.100 ready
3969 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s2 ready
reserved:host
Stderr:
===================== Exiting AfterFailed =====================
21:57:13 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
21:57:13 STEP: Deleting deployment demo_ds.yaml
21:57:15 STEP: Deleting namespace 202109102156k8sdatapathconfigautodirectnoderoutescheckconnectiv
21:57:31 STEP: Running AfterEach for block EntireTestsuite
[[ATTACHMENT|2d69cad5_K8sDatapathConfig_AutoDirectNodeRoutes_Check_connectivity_with_sockops_and_direct_routing.zip]]
ZIP Links:
Click to show.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-5.4//976/artifact/2d69cad5_K8sDatapathConfig_AutoDirectNodeRoutes_Check_connectivity_with_sockops_and_direct_routing.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-5.4//976/artifact/test_results_Cilium-PR-K8s-1.19-kernel-5.4_976_BDD-Test-PR.zip
Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-5.4/976/
If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.