CI: Conformance E2E: client-egress-l7-named-port/pod-to-pod: command terminated with exit code 28 (timeout) #27762
Description
CI failure
Matrix entry: (5, 5.15-20230420.212204, iptables, true, disabled, dsr, true, true, true)
ℹ️ 📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
ℹ️ 📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
[-] Scenario [client-egress-l7-named-port/pod-to-pod]
[.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-0: cilium-test/client-6b4b857d98-hnf49 (10.244.1.76) -> cilium-test/echo-other-node-8f5d547bf-sgh62 (10.244.2.69:8080)]
[.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-0: cilium-test/client-6b4b857d98-hnf49 (fd00:10:244:1::47c6) -> cilium-test/echo-other-node-8f5d547bf-sgh62 (fd00:10:244:2::294e:8080)]
[.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-1: cilium-test/client-6b4b857d98-hnf49 (10.244.1.76) -> cilium-test/echo-same-node-7c978647dc-xr2kt (10.244.1.111:8080)]
[.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-1: cilium-test/client-6b4b857d98-hnf49 (fd00:10:244:1::47c6) -> cilium-test/echo-same-node-7c978647dc-xr2kt (fd00:10:244:1::19cd:8080)]
[.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-2: cilium-test/client2-646b88fb9b-x8j2f (10.244.1.44) -> cilium-test/echo-other-node-8f5d547bf-sgh62 (10.244.2.69:8080)]
❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://10.244.2.69:8080/" failed: command terminated with exit code 28
ℹ️ curl output:
📋 Test Report
connectivity test failed: 1 tests failed
❌ 1/53 tests failed (1/575 actions), 8 tests skipped, 0 scenarios skipped:
Test [client-egress-l7-named-port]:
❌ client-egress-l7-named-port/pod-to-pod/curl-ipv4-2: cilium-test/client2-646b88fb9b-x8j2f (10.244.1.44) -> cilium-test/echo-other-node-8f5d547bf-sgh62 (10.244.2.69:8080)
Error: Process completed with exit code 1.
Looks similar to #27672 but not related to multicluster.
From a quick sysdump check, there seems to be something wrong with Envoy and the http conneciton:
$ cat hubble-flows-cilium-* | hubble observe --input-file - --numeric --to-ip 10.244.2.69 --port 8080 --from-ip 10.244.1.44 --type l7
Aug 28 17:16:59.869: 10.244.1.44:44528 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/)
Aug 28 17:17:21.203: 10.244.1.44:54076 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/)
Aug 28 17:15:50.297: 10.244.1.44:37878 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/public)
Aug 28 17:15:50.377: 10.244.1.44:45332 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request DROPPED (HTTP/1.1 GET http://10.244.2.69:8080/private)
Aug 28 17:15:50.453: 10.244.1.44:45348 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/private)
Aug 28 17:16:18.302: 10.244.1.44:45148 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/public)
Aug 28 17:16:18.375: 10.244.1.44:45162 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request DROPPED (HTTP/1.1 GET http://10.244.2.69:8080/private)
Aug 28 17:16:18.445: 10.244.1.44:45166 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/private)