Bug report

General Information

I'm running DNS queries from the Cilium agent pod (192.168.146.198) against CoreDNS running on a remote node. (10.0.1.120) nslookup times out.

13:42:43.672847 IP 192.168.146.198.49128 > 10.0.1.120.53: 11112+ AAAA? jenkins.cilium.io. (35)
13:42:43.673706 IP 192.168.186.218.53 > 192.168.146.198.49128: 11112* 0/1/0 (112)

The reply packet is incorrectly SNATed to the source IP of the other node (192.168.186.218), breaking the connection.

Note that this breaks the FQDN proxy in tunnel mode because DNS lookups are performed from the agent pod.

• Cilium version: 1.10.3
• Kernel version: 5.4.129-62.227.amzn2.x86_64
• Sysdump attached
  cilium-sysdump-20210722-164747.zip

How to reproduce the issue

1. Set up EKS cluster
2. ./cilium install --datapath-mode=tunnel --ipam cluster-pool
3. Drop into Cilium agent pod and nslookup against a CoreDNS instance on another node.

Possibly related:

• CI: ConformanceEKS (ci-eks): DNS failure (RCode: Server Failure) #16963
• ConformanceEKS (ci-eks) CI: no-policies/pod-to-world/http-to-jenkins-cilium: Fails during connectivity test after deploying with encryption enabled #16938

After fixing the issue

• Test exceptions added to EKS (Tunnel) workflow in cilium-cli by workflows: add test exceptions to EKS tunnel workflow cilium-cli#450 need to be removed.