Skip to content

CI: ConformanceKind1.19: connectivity test failed: 5 tests failed: Connectivity to 1.1.1.1 is broken #22217

@joestringer

Description

@joestringer

The ConformanceKind workflow seems to be unstable on master while connecting to 1.1.1.1.

Example run:
https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960
cilium-sysdump-out.zip (28).zip

Run cilium connectivity test --flow-validation=disabled --hubble=false
  cilium connectivity test --flow-validation=disabled --hubble=false
  shell: /usr/bin/bash -e {0}
  env:
    kind_version: v0.11.1
    kind_config: .github/kind-config.yaml
    cilium_cli_version: v0.1[2](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:2).4
    QUAY_ORGANIZATION: cilium
    QUAY_ORGANIZATION_DEV: cilium
✨ [kind-chart-testing] Creating namespace cilium-test for connectivity check...
✨ [kind-chart-testing] Deploying echo-same-node service...
✨ [kind-chart-testing] Deploying DNS test server configmap...
✨ [kind-chart-testing] Deploying same-node deployment...
✨ [kind-chart-testing] Deploying client deployment...
✨ [kind-chart-testing] Deploying client2 deployment...
✨ [kind-chart-testing] Deploying echo-other-node service...
✨ [kind-chart-testing] Deploying other-node deployment...
⌛ [kind-chart-testing] Waiting for deployments [client client2 echo-same-node] to become ready...
⌛ [kind-chart-testing] Waiting for deployments [echo-other-node] to become ready...
⌛ [kind-chart-testing] Waiting for CiliumEndpoint for pod cilium-test/client-7bdbddd7b-bc67l to appear...
⌛ [kind-chart-testing] Waiting for CiliumEndpoint for pod cilium-test/client2-6ff4656b7d-wjj8j to appear...
⌛ [kind-chart-testing] Waiting for pod cilium-test/client-7bdbddd7b-bc67l to reach DNS server on cilium-test/echo-same-node-77494b4cbd-shdlv pod...
⌛ [kind-chart-testing] Waiting for pod cilium-test/client2-6ff4656b7d-wjj8j to reach DNS server on cilium-test/echo-same-node-77494b4cbd-shdlv pod...
⌛ [kind-chart-testing] Waiting for pod cilium-test/client-7bdbddd7b-bc67l to reach DNS server on cilium-test/echo-other-node-c7cd7697d-sqdsb pod...
⌛ [kind-chart-testing] Waiting for pod cilium-test/client2-6ff4656b7d-wjj8j to reach DNS server on cilium-test/echo-other-node-c7cd7697d-sqdsb pod...
⌛ [kind-chart-testing] Waiting for pod cilium-test/client-7bdbddd7b-bc67l to reach default/kubernetes service...
⌛ [kind-chart-testing] Waiting for pod cilium-test/client2-6ff4656b7d-wjj8j to reach default/kubernetes service...
⌛ [kind-chart-testing] Waiting for CiliumEndpoint for pod cilium-test/echo-other-node-c7cd7697d-sqdsb to appear...
⌛ [kind-chart-testing] Waiting for CiliumEndpoint for pod cilium-test/echo-same-node-77494b4cbd-shdlv to appear...
⌛ [kind-chart-testing] Waiting for Service cilium-test/echo-other-node to become ready...
⌛ [kind-chart-testing] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [kind-chart-testing] Waiting for NodePort 172.18.0.2:[3](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:3)2177 (cilium-test/echo-other-node) to become ready...
⌛ [kind-chart-testing] Waiting for NodePort 172.18.0.2:30292 (cilium-test/echo-same-node) to become ready...
⌛ [kind-chart-testing] Waiting for NodePort 172.18.0.3:32177 (cilium-test/echo-other-node) to become ready...
⌛ [kind-chart-testing] Waiting for NodePort 172.18.0.3:30292 (cilium-test/echo-same-node) to become ready...
ℹ️  Skipping IPCache check
ℹ️  Cilium version: 1.12.90
🏃 Running tests...

[=] Test [no-policies]
...................................
  [-] Scenario [no-policies/pod-to-pod]
  [.] Action [no-policies/pod-to-pod/curl-0: cilium-test/client2-6ff[4](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:4)6[5](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:5)[6](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:6)b[7](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:7)d-wjj[8](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:8)j (10.244.1.237) -> cilium-test/echo-other-node-c7cd76[9](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:9)7d-sqdsb ([10](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:11).244.0.57:8080)]
  [.] Action [no-policies/pod-to-pod/curl-1: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> cilium-test/echo-same-node-77494b4cbd-shdlv (10.244.1.233:8080)]
  [.] Action [no-policies/pod-to-pod/curl-2: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cilium-test/echo-other-node-c7cd7697d-sqdsb (10.244.0.57:8080)]
  [.] Action [no-policies/pod-to-pod/curl-3: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cilium-test/echo-same-node-77494b4cbd-shdlv (10.244.1.233:8080)]
  [-] Scenario [no-policies/pod-to-hostport]
  [.] Action [no-policies/pod-to-hostport/curl-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cilium-test/echo-same-node-77494b4cbd-shdlv (172.18.0.3:40000)]
  [.] Action [no-policies/pod-to-hostport/curl-1: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cilium-test/echo-other-node-c7cd7697d-sqdsb (172.18.0.2:40000)]
  [-] Scenario [client-egress-to-cidr-deny/pod-to-cidr]
  [.] Action [client-egress-to-cidr-deny/pod-to-cidr/cloudflare-1001-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cloudflare-1001 (1.0.0.1:80)]
  [.] Action [client-egress-to-cidr-deny/pod-to-cidr/cloudflare-1001-1: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> cloudflare-1001 (1.0.0.1:80)]
  [.] Action [client-egress-to-cidr-deny/pod-to-cidr/cloudflare-[11](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:12)11-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cloudflare-1111 (1.1.1.1:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://1.1.1.1:80" failed: command terminated with exit code 28
  ℹ️  curl output:
  curl: (28) Connection timeout after 5000 ms
:0 -> :0 = 000
  
  📄 No flows recorded during action cloudflare-1111-0
  📄 No flows recorded during action cloudflare-1111-0
  [.] Action [client-egress-to-cidr-deny/pod-to-cidr/cloudflare-1111-1: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> cloudflare-1111 (1.1.1.1:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://1.1.1.1:80" failed: command terminated with exit code 28
  ℹ️  curl output:
  curl: (28) Connection timeout after 5000 ms
:0 -> :0 = 000
  
  📄 No flows recorded during action cloudflare-1111-1
  📄 No flows recorded during action cloudflare-1111-1
  ℹ️  📜 Deleting CiliumNetworkPolicy 'allow-all-egress' from namespace 'cilium-test'..
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-to-cidr-deny' from namespace 'cilium-test'..

[=] Test [health]
..
[=] Test [client-egress-l7-method]
............
[=] Test [client-egress-l7]
..........
[=] Test [client-egress-l7-named-port]
........
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  [-] Scenario [client-egress-l7-named-port/pod-to-pod]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cilium-test/echo-other-node-c7cd7697d-sqdsb (10.244.0.57:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-1: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cilium-test/echo-same-node-77494b4cbd-shdlv (10.244.1.233:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-2: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> cilium-test/echo-other-node-c7cd7697d-sqdsb (10.244.0.57:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-3: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> cilium-test/echo-same-node-77494b4cbd-shdlv (10.244.1.233:8080)]
  [-] Scenario [client-egress-l7-named-port/pod-to-world]
  [.] Action [client-egress-l7-named-port/pod-to-world/http-to-one-one-one-one-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> one-one-one-one-http (one.one.one.one:80)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-one-one-one-one-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> one-one-one-one-https (one.one.one.one:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> one-one-one-one-https-index (one.one.one.one:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed: command terminated with exit code [22](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:23)
  ℹ️  curl output:
  curl: (22) The requested URL returned error: 503
10.244.1.[23](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:24)7:52666 -> 1.1.1.1:80 = 503
  
  📄 No flows recorded during action http-to-one-one-one-one-1
  📄 No flows recorded during action http-to-one-one-one-one-1
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-one-one-one-one-1: cilium-test/client2-6ff4656b7d-wjj8j (10.[24](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:25)4.1.237) -> one-one-one-one-https (one.one.one.one:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> one-one-one-one-https-index (one.one.one.one:443)]
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-only-dns' from namespace 'cilium-test'..
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-l7-http-named-port' from namespace 'cilium-test'..

[=] Test [dns-only]
..........
[=] Test [to-fqdns]
........
📋 Test Report
connectivity test failed: 5 tests failed
❌ 5/30 tests failed (9/2[28](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:29) actions), 0 tests skipped, 0 scenarios skipped:
Test [no-policies]:
  ❌ no-policies/pod-to-cidr/cloudflare-1111-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cloudflare-1111 (1.1.1.1:80)
  ❌ no-policies/pod-to-cidr/cloudflare-1111-1: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.2[37](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:38)) -> cloudflare-1111 (1.1.1.1:80)
Test [all-ingress-deny]:
  ❌ all-ingress-deny/pod-to-cidr/cloudflare-1111-0: cilium-test/client2-6ff4656b7d-wjj8j (10.2[44](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:45).1.237) -> cloudflare-1111 (1.1.1.1:80)
  ❌ all-ingress-deny/pod-to-cidr/cloudflare-1111-1: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cloudflare-1111 (1.1.1.1:80)
Test [to-cidr-1111]:
  ❌ to-cidr-1111/pod-to-cidr/cloudflare-1111-0: cilium-test/client-7bdbddd7b-bc67l (10.244.1.97) -> cloudflare-1111 (1.1.1.1:80)
  ❌ to-cidr-1111/pod-to-cidr/cloudflare-1111-1: cilium-test/client2-6ff[46](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:47)56b7d-wjj8j (10.244.1.[237](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:238)) -> cloudflare-1111 (1.1.1.1:80)
Test [client-egress-to-cidr-deny]:
  ❌ client-egress-to-cidr-deny/pod-to-cidr/cloudflare-1111-0: cilium-test/client-7bdbddd7b-bc67l (10.[244](https://github.com/cilium/cilium/actions/runs/3468759658/jobs/5794941960#step:12:245).1.97) -> cloudflare-1111 (1.1.1.1:80)
  ❌ client-egress-to-cidr-deny/pod-to-cidr/cloudflare-1111-1: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> cloudflare-1111 (1.1.1.1:80)
Test [client-egress-l7-named-port]:
  ❌ client-egress-l7-named-port/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6ff4656b7d-wjj8j (10.244.1.237) -> one-one-one-one-http (one.one.one.one:80)
Error: Process completed with exit code 1.

Metadata

Metadata

Assignees

Labels

area/CIContinuous Integration testing issue or flakeci/flakeThis is a known failure that occurs in the tree. Please investigate me!

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions