CI: K8sPolicyTest Basic Test Denies traffic with k8s default-deny ingress-egress policy #17466
Description
Happened once but for a totally unrelated PR (skipping a test): https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/1435/testReport/junit/Suite-k8s-1/21/K8sPolicyTest_Basic_Test_Denies_traffic_with_k8s_default_deny_ingress_egress_policy/
54df0431_K8sPolicyTest_Basic_Test_Denies_traffic_with_k8s_default-deny_ingress-egress_policy.zip
Looks similar to #17257.
Stacktrace
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Egress ping connectivity should be denied for pod "app2-58757b7dd5-wdzsr"
Expected command: kubectl exec -n 202109210850k8spolicytestbasictestchecksallkindofkubernetespoli app2-58757b7dd5-wdzsr -- ping -W 5 -c 5 8.8.8.8
To have failed, but it was successful:
Exitcode: 0
Stdout:
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4188ms
Stderr:
/home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9/src/github.com/cilium/cilium/test/k8sT/Policies.go:768
Standard Output
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: [cilium-gvncc cilium-hm9p5]
Netpols loaded: 202109210850k8spolicytestbasictestchecksallkindofkubernetespoli::knp-default-deny-ingress-egress
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
Pod Ingress Egress
app3-5d69599cdd-hx7nh
coredns-755cd654d4-r7rng
app1-7469cfcb66-q97d2
app1-7469cfcb66-ssfq4
app2-58757b7dd5-wdzsr
Cilium agent 'cilium-gvncc': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 38 Failed 0
Cilium agent 'cilium-hm9p5': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 18 Failed 0
Standard Error
Click to show
08:54:38 STEP: Running BeforeEach block for EntireTestsuite K8sPolicyTest Basic Test
08:54:41 STEP: WaitforPods(namespace="202109210850k8spolicytestbasictestchecksallkindofkubernetespoli", filter="-l zgroup=testapp")
08:54:41 STEP: WaitforPods(namespace="202109210850k8spolicytestbasictestchecksallkindofkubernetespoli", filter="-l zgroup=testapp") => <nil>
08:54:41 STEP: Installing knp ingress-egress default-deny
08:54:45 STEP: Testing if egress and ingress policy enforcement is enabled on the endpoint
FAIL: Egress ping connectivity should be denied for pod "app2-58757b7dd5-wdzsr"
Expected command: kubectl exec -n 202109210850k8spolicytestbasictestchecksallkindofkubernetespoli app2-58757b7dd5-wdzsr -- ping -W 5 -c 5 8.8.8.8
To have failed, but it was successful:
Exitcode: 0
Stdout:
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4188ms
Stderr:
=== Test Finished at 2021-09-21T08:55:01Z====
08:55:01 STEP: Running JustAfterEach block for EntireTestsuite K8sPolicyTest
===================== TEST FAILED =====================
08:55:01 STEP: Running AfterFailed block for EntireTestsuite K8sPolicyTest
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
202109210850k8spolicytestbasictestchecksallkindofkubernetespoli app1-7469cfcb66-q97d2 2/2 Running 0 5m3s 10.0.1.222 k8s1 <none> <none>
202109210850k8spolicytestbasictestchecksallkindofkubernetespoli app1-7469cfcb66-ssfq4 2/2 Running 0 5m3s 10.0.1.251 k8s1 <none> <none>
202109210850k8spolicytestbasictestchecksallkindofkubernetespoli app2-58757b7dd5-wdzsr 1/1 Running 0 5m3s 10.0.1.186 k8s1 <none> <none>
202109210850k8spolicytestbasictestchecksallkindofkubernetespoli app3-5d69599cdd-hx7nh 1/1 Running 0 5m3s 10.0.1.41 k8s1 <none> <none>
cilium-monitoring grafana-5747bcc8f9-v7sv2 0/1 Running 0 13m 10.0.0.242 k8s2 <none> <none>
cilium-monitoring prometheus-655fb888d7-h7z56 1/1 Running 0 13m 10.0.0.97 k8s2 <none> <none>
kube-system cilium-gvncc 1/1 Running 0 6m16s 192.168.36.11 k8s1 <none> <none>
kube-system cilium-hm9p5 1/1 Running 0 6m16s 192.168.36.12 k8s2 <none> <none>
kube-system cilium-operator-6c84c4777-grtnx 1/1 Running 0 6m15s 192.168.36.12 k8s2 <none> <none>
kube-system cilium-operator-6c84c4777-v44jg 1/1 Running 0 6m15s 192.168.36.11 k8s1 <none> <none>
kube-system coredns-755cd654d4-r7rng 1/1 Running 0 5m14s 10.0.1.226 k8s1 <none> <none>
kube-system etcd-k8s1 1/1 Running 0 16m 192.168.36.11 k8s1 <none> <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 16m 192.168.36.11 k8s1 <none> <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 16m 192.168.36.11 k8s1 <none> <none>
kube-system kube-proxy-vpj29 1/1 Running 0 14m 192.168.36.12 k8s2 <none> <none>
kube-system kube-proxy-vzf22 1/1 Running 0 16m 192.168.36.11 k8s1 <none> <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 16m 192.168.36.11 k8s1 <none> <none>
kube-system log-gatherer-4456z 1/1 Running 0 13m 192.168.36.12 k8s2 <none> <none>
kube-system log-gatherer-x798m 1/1 Running 0 13m 192.168.36.11 k8s1 <none> <none>
kube-system registry-adder-2hst2 1/1 Running 0 14m 192.168.36.11 k8s1 <none> <none>
kube-system registry-adder-mvtwx 1/1 Running 0 14m 192.168.36.12 k8s2 <none> <none>
Stderr:
Fetching command output from pods [cilium-gvncc cilium-hm9p5]
cmd: kubectl exec -n kube-system cilium-gvncc -c cilium-agent -- cilium service list
Exitcode: 0
Stdout:
ID Frontend Service Type Backend
1 10.96.0.1:443 ClusterIP 1 => 192.168.36.11:6443
2 10.96.0.10:53 ClusterIP 1 => 10.0.1.226:53
3 10.96.0.10:9153 ClusterIP 1 => 10.0.1.226:9153
4 10.102.167.119:3000 ClusterIP
5 10.109.182.228:9090 ClusterIP 1 => 10.0.0.97:9090
6 10.111.146.180:80 ClusterIP 1 => 10.0.1.222:80
2 => 10.0.1.251:80
7 10.111.146.180:69 ClusterIP 1 => 10.0.1.222:69
2 => 10.0.1.251:69
Stderr:
cmd: kubectl exec -n kube-system cilium-gvncc -c cilium-agent -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
9 Enabled Enabled 20141 k8s:id=app3 fd02::12b 10.0.1.41 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202109210850k8spolicytestbasictestchecksallkindofkubernetespoli
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202109210850k8spolicytestbasictestchecksallkindofkubernetespoli
k8s:zgroup=testapp
46 Enabled Enabled 5786 k8s:appSecond=true fd02::1af 10.0.1.186 ready
k8s:id=app2
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202109210850k8spolicytestbasictestchecksallkindofkubernetespoli
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=app2-account
k8s:io.kubernetes.pod.namespace=202109210850k8spolicytestbasictestchecksallkindofkubernetespoli
k8s:zgroup=testapp
125 Enabled Enabled 11762 k8s:id=app1 fd02::1fa 10.0.1.251 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202109210850k8spolicytestbasictestchecksallkindofkubernetespoli
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=app1-account
k8s:io.kubernetes.pod.namespace=202109210850k8spolicytestbasictestchecksallkindofkubernetespoli
k8s:zgroup=testapp
142 Disabled Disabled 63421 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system fd02::1f0 10.0.1.226 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
1132 Disabled Disabled 4 reserved:health fd02::14c 10.0.1.65 ready
1447 Enabled Enabled 11762 k8s:id=app1 fd02::1d1 10.0.1.222 ready
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=202109210850k8spolicytestbasictestchecksallkindofkubernetespoli
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=app1-account
k8s:io.kubernetes.pod.namespace=202109210850k8spolicytestbasictestchecksallkindofkubernetespoli
k8s:zgroup=testapp
4094 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s1 ready
k8s:node-role.kubernetes.io/control-plane
k8s:node-role.kubernetes.io/master
k8s:node.kubernetes.io/exclude-from-external-load-balancers
reserved:host
Stderr:
cmd: kubectl exec -n kube-system cilium-hm9p5 -c cilium-agent -- cilium service list
Exitcode: 0
Stdout:
ID Frontend Service Type Backend
1 10.96.0.1:443 ClusterIP 1 => 192.168.36.11:6443
2 10.96.0.10:53 ClusterIP 1 => 10.0.1.226:53
3 10.96.0.10:9153 ClusterIP 1 => 10.0.1.226:9153
4 10.102.167.119:3000 ClusterIP
5 10.109.182.228:9090 ClusterIP 1 => 10.0.0.97:9090
6 10.111.146.180:80 ClusterIP 1 => 10.0.1.222:80
2 => 10.0.1.251:80
7 10.111.146.180:69 ClusterIP 1 => 10.0.1.222:69
2 => 10.0.1.251:69
Stderr:
cmd: kubectl exec -n kube-system cilium-hm9p5 -c cilium-agent -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
55 Disabled Disabled 1 k8s:cilium.io/ci-node=k8s2 ready
reserved:host
2756 Disabled Disabled 4 reserved:health fd02::82 10.0.0.139 ready
Stderr:
===================== Exiting AfterFailed =====================
08:55:22 STEP: Running AfterEach for block EntireTestsuite K8sPolicyTest Basic Test
08:55:22 STEP: Running AfterEach for block EntireTestsuite K8sPolicyTest
08:55:22 STEP: Running AfterEach for block EntireTestsuite