CI: K8sDatapathConfig Host firewall With native routing: Managed to reach #15575
Description
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.19/75/testReport/junit/Suite-k8s-1/20/K8sDatapathConfig_Host_firewall_With_native_routing/
8fab5f05_K8sDatapathConfig_Host_firewall_With_native_routing.zip
There are other ongoing flakes for this test, so please check for the Managed to reach part before assuming it's the same flake.
Stacktrace
/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Managed to reach 10.0.1.15:69 from testclient-host-8m2gz
Expected command: kubectl exec -n 202104061126k8sdatapathconfighostfirewallwithnativerouting testclient-host-8m2gz -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 tftp://10.0.1.15:69/hello -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'"
To have failed, but it was successful:
Exitcode: 0
Stdout:
Hostname: testserver-pgqr4
Request Information:
client_address=192.168.36.12
client_port=33119
real path=/hello
request_scheme=tftp
time-> DNS: '0.000015()', Connect: '0.000032',Transfer '0.000000', total '0.001360'
Stderr:
/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:683
Standard Output
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 0
⚠️ Number of "level=warning" in logs: 7
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Disabling HostServicesPeer feature.
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Unable to update ipcache map entry on pod add
Cilium pods: [cilium-f7c5z cilium-mbpff]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
Pod Ingress Egress
testserver-pgqr4
grafana-d69c97b9b-2s6kl
prometheus-655fb888d7-bkd2j
coredns-867bf6789f-ckr7m
testclient-4g9z2
testclient-td67l
testserver-gln6g
Cilium agent 'cilium-f7c5z': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 43 Failed 0
Cilium agent 'cilium-mbpff': Status: Ok Health: Ok Nodes "" ContinerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 29 Failed 0
Standard Error
Click here to see
11:25:29 STEP: Installing Cilium
11:25:31 STEP: Waiting for Cilium to become ready
11:26:10 STEP: Validating if Kubernetes DNS is deployed
11:26:10 STEP: Checking if deployment is ready
11:26:10 STEP: Checking if kube-dns service is plumbed correctly
11:26:10 STEP: Checking if pods have identity
11:26:10 STEP: Checking if DNS can resolve
11:26:16 STEP: Kubernetes DNS is not ready: 5s timeout expired
11:26:16 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
11:26:17 STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mbpff: unable to find service backend 10.0.1.118:53 in datapath of cilium pod cilium-mbpff
11:26:30 STEP: Waiting for Kubernetes DNS to become operational
11:26:30 STEP: Checking if deployment is ready
11:26:30 STEP: Checking if kube-dns service is plumbed correctly
11:26:30 STEP: Checking if pods have identity
11:26:30 STEP: Checking if DNS can resolve
11:26:31 STEP: Validating Cilium Installation
11:26:31 STEP: Performing Cilium controllers preflight check
11:26:31 STEP: Performing Cilium health check
11:26:31 STEP: Performing Cilium status preflight check
11:26:34 STEP: Performing Cilium service preflight check
11:26:34 STEP: Performing K8s service preflight check
11:26:35 STEP: Waiting for cilium-operator to be ready
11:26:35 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
11:26:35 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
11:26:35 STEP: Making sure all endpoints are in ready state
11:26:36 STEP: Creating namespace 202104061126k8sdatapathconfighostfirewallwithnativerouting
11:26:36 STEP: Deploying demo_hostfw.yaml in namespace 202104061126k8sdatapathconfighostfirewallwithnativerouting
11:26:36 STEP: Waiting for 4m0s for 8 pods of deployment demo_hostfw.yaml to become ready
11:26:36 STEP: WaitforNPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="")
11:26:46 STEP: WaitforNPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="") => <nil>
11:26:46 STEP: Applying policies /home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/manifests/host-policies.yaml
11:26:59 STEP: Checking host policies on ingress from local pod
11:26:59 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClient")
11:26:59 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClient") => <nil>
11:26:59 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServerHost")
11:26:59 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServerHost") => <nil>
11:27:06 STEP: Checking host policies on ingress from remote pod
11:27:06 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClient")
11:27:06 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClient") => <nil>
11:27:06 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServerHost")
11:27:06 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServerHost") => <nil>
11:27:12 STEP: Checking host policies on egress to local pod
11:27:12 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClientHost")
11:27:12 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClientHost") => <nil>
11:27:12 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServer")
11:27:12 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServer") => <nil>
11:27:18 STEP: Checking host policies on egress to remote pod
11:27:18 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClientHost")
11:27:18 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testClientHost") => <nil>
11:27:18 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServer")
11:27:18 STEP: WaitforPods(namespace="202104061126k8sdatapathconfighostfirewallwithnativerouting", filter="-l zgroup=testServer") => <nil>
FAIL: Managed to reach 10.0.1.15:69 from testclient-host-8m2gz
Expected command: kubectl exec -n 202104061126k8sdatapathconfighostfirewallwithnativerouting testclient-host-8m2gz -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 tftp://10.0.1.15:69/hello -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'"
To have failed, but it was successful:
Exitcode: 0
Stdout:
Hostname: testserver-pgqr4
Request Information:
client_address=192.168.36.12
client_port=33119
real path=/hello
request_scheme=tftp
time-> DNS: '0.000015()', Connect: '0.000032',Transfer '0.000000', total '0.001360'
Stderr:
=== Test Finished at 2021-04-06T11:27:18Z====
11:27:18 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
11:27:18 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
202104061126k8sdatapathconfighostfirewallwithnativerouting testclient-4g9z2 1/1 Running 0 45s 10.0.1.87 k8s1 <none> <none>
202104061126k8sdatapathconfighostfirewallwithnativerouting testclient-host-8m2gz 1/1 Running 0 45s 192.168.36.12 k8s2 <none> <none>
202104061126k8sdatapathconfighostfirewallwithnativerouting testclient-host-rtslw 1/1 Running 0 45s 192.168.36.11 k8s1 <none> <none>
202104061126k8sdatapathconfighostfirewallwithnativerouting testclient-td67l 1/1 Running 0 45s 10.0.0.152 k8s2 <none> <none>
202104061126k8sdatapathconfighostfirewallwithnativerouting testserver-gln6g 2/2 Running 0 45s 10.0.0.17 k8s2 <none> <none>
202104061126k8sdatapathconfighostfirewallwithnativerouting testserver-host-758rd 2/2 Running 0 45s 192.168.36.12 k8s2 <none> <none>
202104061126k8sdatapathconfighostfirewallwithnativerouting testserver-host-vm4hm 2/2 Running 0 45s 192.168.36.11 k8s1 <none> <none>
202104061126k8sdatapathconfighostfirewallwithnativerouting testserver-pgqr4 2/2 Running 0 45s 10.0.1.15 k8s1 <none> <none>
cilium-monitoring grafana-d69c97b9b-2s6kl 1/1 Running 0 73m 10.0.0.23 k8s2 <none> <none>
cilium-monitoring prometheus-655fb888d7-bkd2j 1/1 Running 0 73m 10.0.0.242 k8s2 <none> <none>
kube-system cilium-f7c5z 1/1 Running 0 110s 192.168.36.12 k8s2 <none> <none>
kube-system cilium-mbpff 1/1 Running 0 110s 192.168.36.11 k8s1 <none> <none>
kube-system cilium-operator-76c8b94696-9qcjs 1/1 Running 0 110s 192.168.36.12 k8s2 <none> <none>
kube-system cilium-operator-76c8b94696-qldvc 1/1 Running 0 110s 192.168.36.11 k8s1 <none> <none>
kube-system coredns-867bf6789f-ckr7m 1/1 Running 0 65s 10.0.0.18 k8s2 <none> <none>
kube-system etcd-k8s1 1/1 Running 0 76m 192.168.36.11 k8s1 <none> <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 76m 192.168.36.11 k8s1 <none> <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 76m 192.168.36.11 k8s1 <none> <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 76m 192.168.36.11 k8s1 <none> <none>
kube-system log-gatherer-9jsrs 1/1 Running 0 74m 192.168.36.11 k8s1 <none> <none>
kube-system log-gatherer-m9bgx 1/1 Running 0 74m 192.168.36.12 k8s2 <none> <none>
kube-system registry-adder-2n9dj 1/1 Running 0 74m 192.168.36.11 k8s1 <none> <none>
kube-system registry-adder-sdfn9 1/1 Running 0 74m 192.168.36.12 k8s2 <none> <none>
Stderr:
Fetching command output from pods [cilium-f7c5z cilium-mbpff]
cmd: kubectl exec -n kube-system cilium-f7c5z -- cilium status
Exitcode: 0
Stdout:
KVStore: Ok Disabled
Kubernetes: Ok 1.20 (v1.20.5) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [enp0s8 192.168.36.12 fd04::12 (Direct Routing), enp0s3 10.0.2.15 fd04::12]
Cilium: Ok 1.9.90 (v.1.9.90-r.2d6fdc4)
NodeMonitor: Listening for events on 3 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 7/255 allocated from 10.0.0.0/24, IPv6: 7/255 allocated from fd02::/120
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: BPF [enp0s8, enp0s3] 10.0.0.0/8 [IPv4: Enabled, IPv6: Enabled]
Controller Status: 43/43 healthy
Proxy Status: OK, ip 10.0.0.45, 0 redirects active on ports 10000-20000
Hubble: Ok Current/Max Flows: 1044/4095 (25.49%), Flows/s: 11.51 Metrics: Disabled
Cluster health: 2/2 reachable (2021-04-06T11:26:33Z)
Stderr:
cmd: kubectl exec -n kube-system cilium-f7c5z -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
36 Disabled Disabled 9202 k8s:io.cilium.k8s.policy.cluster=default fd02::31 10.0.0.18 ready
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
208 Disabled Disabled 8064 k8s:io.cilium.k8s.policy.cluster=default fd02::a 10.0.0.17 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202104061126k8sdatapathconfighostfirewallwithnativerouting
k8s:zgroup=testServer
817 Disabled Disabled 10448 k8s:io.cilium.k8s.policy.cluster=default fd02::ea 10.0.0.152 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202104061126k8sdatapathconfighostfirewallwithnativerouting
k8s:zgroup=testClient
1251 Disabled Disabled 21320 k8s:app=grafana fd02::73 10.0.0.23 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=cilium-monitoring
2302 Enabled Enabled 1 k8s:cilium.io/ci-node=k8s2 ready
reserved:host
2318 Disabled Disabled 4 reserved:health fd02::1 10.0.0.177 ready
4053 Disabled Disabled 21889 k8s:app=prometheus fd02::75 10.0.0.242 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s
k8s:io.kubernetes.pod.namespace=cilium-monitoring
Stderr:
cmd: kubectl exec -n kube-system cilium-mbpff -- cilium status
Exitcode: 0
Stdout:
KVStore: Ok Disabled
Kubernetes: Ok 1.20 (v1.20.5) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [enp0s8 192.168.36.11 fd04::11 (Direct Routing), enp0s3 10.0.2.15 fd04::11]
Cilium: Ok 1.9.90 (v.1.9.90-r.2d6fdc4)
NodeMonitor: Listening for events on 3 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 4/255 allocated from 10.0.1.0/24, IPv6: 4/255 allocated from fd02::100/120
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: BPF [enp0s8, enp0s3] 10.0.0.0/8 [IPv4: Enabled, IPv6: Enabled]
Controller Status: 29/29 healthy
Proxy Status: OK, ip 10.0.1.35, 0 redirects active on ports 10000-20000
Hubble: Ok Current/Max Flows: 1338/4095 (32.67%), Flows/s: 16.06 Metrics: Disabled
Cluster health: 2/2 reachable (2021-04-06T11:27:15Z)
Stderr:
cmd: kubectl exec -n kube-system cilium-mbpff -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
399 Disabled Disabled 8064 k8s:io.cilium.k8s.policy.cluster=default fd02::15e 10.0.1.15 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202104061126k8sdatapathconfighostfirewallwithnativerouting
k8s:zgroup=testServer
767 Disabled Disabled 10448 k8s:io.cilium.k8s.policy.cluster=default fd02::138 10.0.1.87 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=202104061126k8sdatapathconfighostfirewallwithnativerouting
k8s:zgroup=testClient
987 Disabled Disabled 4 reserved:health fd02::1c7 10.0.1.222 ready
2160 Enabled Enabled 1 k8s:cilium.io/ci-node=k8s1 ready
k8s:node-role.kubernetes.io/control-plane
k8s:node-role.kubernetes.io/master
reserved:host
Stderr:
===================== Exiting AfterFailed =====================
11:27:43 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig Host firewall
11:27:43 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
11:27:43 STEP: Deleting deployment demo_hostfw.yaml
11:27:43 STEP: Deleting namespace 202104061126k8sdatapathconfighostfirewallwithnativerouting
11:27:43 STEP: Deleting namespace 202104061126k8sdatapathconfighostfirewallwithnativerouting
11:27:58 STEP: Running AfterEach for block EntireTestsuite