CI: v1.8: K8sServicesTest Checks service across nodes: cannot retrieve pod names by filter "zgroup=testDSClient" #13008
Description
In https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-K8s/3458/testReport/ the following tests failed with the same error message cannot retrieve pod names by filter "zgroup=testDSClient". Happened in backport PR #12990.
Suite-k8s-1.12.K8sServicesTest Checks service across nodes Tests NodePort (kube-proxy)
Suite-k8s-1.12.K8sServicesTest Checks service across nodes with L4 policy Tests NodePort with L4 Policy
Suite-k8s-1.12.K8sServicesTest Checks service across nodes with L7 policy Tests NodePort with L7 Policy
Suite-k8s-1.13.K8sServicesTest Checks service across nodes with L4 policy Tests NodePort with L4 Policy
test_results_Cilium-PR-Ginkgo-Tests-K8s_3458_BDD-Test-k8s-1.12.zip
Stacktrace
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-K8s/1.13-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:514
cannot retrieve pod names by filter "zgroup=testDSClient"
Expected
<*errors.errorString | 0xc000b6bbd0>: {
s: "could not find pods in namespace 'default' with label 'zgroup=testDSClient': context deadline exceeded",
}
to be nil
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-K8s/1.13-gopath/src/github.com/cilium/cilium/test/k8sT/Services.go:736
Standard Output
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: [cilium-99l2z cilium-bkc69]
Netpols loaded:
CiliumNetworkPolicies loaded: default::l4-policy-demo
Endpoint Policy Enforcement:
Pod Ingress Egress
grafana-b4dbb994f-tx5fx
prometheus-688959f59d-djj2b
test-k8s2-749b7c58cc-jxr4q
testclient-kjpvk
testclient-xx9rd
testds-n2h9z
testds-nw5sj
coredns-7ff984754c-g64wh
Cilium agent 'cilium-99l2z': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 48 Failed 0
Cilium agent 'cilium-bkc69': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 28 Failed 0
Standard Error
Show standard error
01:03:58 STEP: Running BeforeAll block for EntireTestsuite K8sServicesTest Checks service across nodes with L4 policy
01:04:00 STEP: Applying policy /home/vagrant/go/src/github.com/cilium/cilium/test/k8sT/manifests/l4-policy-demo.yaml
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "tftp://[::ffff:192.168.36.12]:31018/hello"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "tftp://[::ffff:127.0.0.1]:31018/hello"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "tftp://10.110.178.159:10069/hello"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "tftp://192.168.36.11:31018/hello"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "http://127.0.0.1:30384"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "tftp://127.0.0.1:31018/hello"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "http://[::ffff:127.0.0.1]:30384"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "http://192.168.36.12:30384"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "http://[::ffff:192.168.36.11]:30384"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "http://10.110.178.159:10080"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "tftp://[::ffff:192.168.36.11]:31018/hello"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "http://192.168.36.11:30384"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "http://[::ffff:192.168.36.12]:30384"
01:04:14 STEP: Making 10 curl requests from pod (host netns) k8s1 to "tftp://192.168.36.12:31018/hello"
01:04:20 STEP: Making 10 curl requests from testclient-kjpvk pod to service tftp://10.110.178.159:10069/hello
01:04:20 STEP: Making 10 curl requests from testclient-kjpvk pod to service tftp://[::ffff:192.168.36.11]:31018/hello
01:04:20 STEP: Making 10 curl requests from testclient-kjpvk pod to service tftp://192.168.36.11:31018/hello
01:04:21 STEP: Making 10 curl requests from testclient-kjpvk pod to service http://[::ffff:192.168.36.11]:30384
01:04:21 STEP: Making 10 curl requests from testclient-kjpvk pod to service http://[::ffff:192.168.36.12]:30384
01:04:21 STEP: Making 10 curl requests from testclient-kjpvk pod to service tftp://[::ffff:192.168.36.12]:31018/hello
01:04:23 STEP: Making 10 curl requests from testclient-kjpvk pod to service http://192.168.36.12:30384
01:04:24 STEP: Making 10 curl requests from testclient-xx9rd pod to service tftp://10.110.178.159:10069/hello
01:04:24 STEP: Making 10 curl requests from testclient-kjpvk pod to service http://10.110.178.159:10080
FAIL: cannot retrieve pod names by filter "zgroup=testDSClient"
Expected
<*errors.errorString | 0xc000b6bbd0>: {
s: "could not find pods in namespace 'default' with label 'zgroup=testDSClient': context deadline exceeded",
}
to be nil
FAIL: cannot retrieve pod names by filter "zgroup=testDSClient"
Expected
<*errors.errorString | 0xc0011014d0>: {
s: "could not find pods in namespace 'default' with label 'zgroup=testDSClient': context deadline exceeded",
}
to be nil
01:04:28 STEP: Making 10 curl requests from testclient-xx9rd pod to service http://[::ffff:192.168.36.12]:30384
01:04:29 STEP: Making 10 curl requests from testclient-xx9rd pod to service tftp://[::ffff:192.168.36.11]:31018/hello
01:04:30 STEP: Making 10 curl requests from testclient-xx9rd pod to service http://[::ffff:192.168.36.11]:30384
01:04:31 STEP: Making 10 curl requests from testclient-xx9rd pod to service tftp://[::ffff:192.168.36.12]:31018/hello
01:04:31 STEP: Making 10 curl requests from testclient-xx9rd pod to service tftp://192.168.36.11:31018/hello
01:04:32 STEP: Making 10 curl requests from testclient-xx9rd pod to service http://10.110.178.159:10080
01:04:32 STEP: Making 10 curl requests from testclient-xx9rd pod to service http://192.168.36.12:30384
=== Test Finished at 2020-08-28T01:04:52Z====
01:04:52 STEP: Running JustAfterEach block for EntireTestsuite K8sServicesTest
===================== TEST FAILED =====================
01:04:54 STEP: Running AfterFailed block for EntireTestsuite K8sServicesTest
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cilium-monitoring grafana-b4dbb994f-tx5fx 1/1 Running 0 12m 10.0.0.143 k8s2 <none> <none>
cilium-monitoring prometheus-688959f59d-djj2b 1/1 Running 0 12m 10.0.0.24 k8s2 <none> <none>
default test-k8s2-749b7c58cc-jxr4q 2/2 Running 0 6m58s 10.0.0.210 k8s2 <none> <none>
default testclient-kjpvk 1/1 Running 0 7m 10.0.0.79 k8s2 <none> <none>
default testclient-xx9rd 1/1 Running 0 7m 10.0.1.194 k8s1 <none> <none>
default testds-n2h9z 2/2 Running 0 7m 10.0.1.83 k8s1 <none> <none>
default testds-nw5sj 2/2 Running 0 7m 10.0.0.104 k8s2 <none> <none>
kube-system cilium-99l2z 1/1 Running 0 3m25s 192.168.36.12 k8s2 <none> <none>
kube-system cilium-bkc69 1/1 Running 0 3m25s 192.168.36.11 k8s1 <none> <none>
kube-system cilium-operator-58bd4497bf-hcgtp 1/1 Running 0 3m25s 192.168.36.12 k8s2 <none> <none>
kube-system coredns-7ff984754c-g64wh 1/1 Running 0 8m24s 10.0.0.106 k8s2 <none> <none>
kube-system etcd-k8s1 1/1 Running 0 20m 192.168.36.11 k8s1 <none> <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 20m 192.168.36.11 k8s1 <none> <none>
kube-system kube-controller-manager-k8s1 0/1 CrashLoopBackOff 1 20m 192.168.36.11 k8s1 <none> <none>
kube-system kube-proxy-9mkh4 1/1 Running 0 21m 192.168.36.11 k8s1 <none> <none>
kube-system kube-proxy-st9md 1/1 Running 0 17m 192.168.36.12 k8s2 <none> <none>
kube-system kube-scheduler-k8s1 0/1 CrashLoopBackOff 1 20m 192.168.36.11 k8s1 <none> <none>
kube-system log-gatherer-2gcwb 1/1 Running 0 12m 192.168.36.12 k8s2 <none> <none>
kube-system log-gatherer-rxfzh 1/1 Running 0 12m 192.168.36.11 k8s1 <none> <none>
kube-system registry-adder-7xp4g 1/1 Running 0 17m 192.168.36.11 k8s1 <none> <none>
kube-system registry-adder-x6qnf 1/1 Running 0 17m 192.168.36.12 k8s2 <none> <none>
Stderr:
Fetching command output from pods [cilium-99l2z cilium-bkc69]
cmd: kubectl exec -n kube-system cilium-99l2z -- cilium service list
Exitcode: 0
Stdout:
ID Frontend Service Type Backend
1 10.96.0.1:443 ClusterIP 1 => 192.168.36.11:6443
2 10.96.0.10:53 ClusterIP 1 => 10.0.0.106:53
3 10.98.200.92:3000 ClusterIP 1 => 10.0.0.143:3000
4 10.99.93.173:9090 ClusterIP 1 => 10.0.0.24:9090
9 10.97.127.175:80 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
10 10.97.127.175:69 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
11 10.110.178.159:10080 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
12 10.110.178.159:10069 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
13 10.98.69.99:10080 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
14 10.98.69.99:10069 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
15 10.103.52.15:10080 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
16 10.103.52.15:10069 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
17 10.101.251.21:10069 ClusterIP 1 => 10.0.0.210:69
18 10.101.251.21:10080 ClusterIP 1 => 10.0.0.210:80
19 10.103.16.186:10080 ClusterIP 1 => 10.0.0.210:80
20 10.103.16.186:10069 ClusterIP 1 => 10.0.0.210:69
21 10.110.35.194:80 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
22 10.105.219.35:80 ClusterIP 1 => 10.0.0.210:80
23 10.97.148.39:20069 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
24 10.97.148.39:20080 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
Stderr:
cmd: kubectl exec -n kube-system cilium-99l2z -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
33 Disabled Disabled 38292 k8s:io.cilium.k8s.policy.cluster=default fd00::49 10.0.0.79 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testDSClient
131 Disabled Disabled 42798 k8s:io.cilium.k8s.policy.cluster=default fd00::b9 10.0.0.106 ready
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
585 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s2 ready
reserved:host
994 Disabled Disabled 3686 k8s:app=prometheus fd00::e6 10.0.0.24 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s
k8s:io.kubernetes.pod.namespace=cilium-monitoring
1891 Disabled Disabled 15202 k8s:app=grafana fd00::11 10.0.0.143 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=cilium-monitoring
2439 Disabled Disabled 4 reserved:health fd00::4d 10.0.0.15 ready
2443 Disabled Disabled 5839 k8s:io.cilium.k8s.policy.cluster=default fd00::a7 10.0.0.210 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=test-k8s2
3432 Enabled Disabled 60312 k8s:io.cilium.k8s.policy.cluster=default fd00::13 10.0.0.104 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testDS
Stderr:
cmd: kubectl exec -n kube-system cilium-bkc69 -- cilium service list
Exitcode: 0
Stdout:
ID Frontend Service Type Backend
1 10.98.200.92:3000 ClusterIP 1 => 10.0.0.143:3000
2 10.99.93.173:9090 ClusterIP 1 => 10.0.0.24:9090
3 10.96.0.1:443 ClusterIP 1 => 192.168.36.11:6443
4 10.96.0.10:53 ClusterIP 1 => 10.0.0.106:53
9 10.97.127.175:69 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
10 10.97.127.175:80 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
11 10.110.178.159:10080 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
12 10.110.178.159:10069 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
13 10.98.69.99:10080 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
14 10.98.69.99:10069 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
15 10.103.52.15:10069 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
16 10.103.52.15:10080 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
17 10.101.251.21:10069 ClusterIP 1 => 10.0.0.210:69
18 10.101.251.21:10080 ClusterIP 1 => 10.0.0.210:80
19 10.103.16.186:10069 ClusterIP 1 => 10.0.0.210:69
20 10.103.16.186:10080 ClusterIP 1 => 10.0.0.210:80
21 10.110.35.194:80 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
22 10.105.219.35:80 ClusterIP 1 => 10.0.0.210:80
23 10.97.148.39:20080 ClusterIP 1 => 10.0.1.83:80
2 => 10.0.0.104:80
24 10.97.148.39:20069 ClusterIP 1 => 10.0.1.83:69
2 => 10.0.0.104:69
Stderr:
cmd: kubectl exec -n kube-system cilium-bkc69 -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
458 Enabled Disabled 60312 k8s:io.cilium.k8s.policy.cluster=default fd00::139 10.0.1.83 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testDS
749 Disabled Disabled 38292 k8s:io.cilium.k8s.policy.cluster=default fd00::1c2 10.0.1.194 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testDSClient
1266 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s1 ready
k8s:node-role.kubernetes.io/master
reserved:host
2516 Disabled Disabled 4 reserved:health fd00::1a5 10.0.1.241 ready
Stderr:
===================== Exiting AfterFailed =====================
01:06:53 STEP: Running AfterEach for block EntireTestsuite K8sServicesTest
01:06:53 STEP: Running AfterEach for block EntireTestsuite