Is there an existing issue for this?

• I have searched the existing issues

What happened?

We observed drop SYN packet in cilium pod of egress gateway, and reason was "No mapping for NAT masquerade".
After received the error message, we show the nat mapping table by filtering a egress ip, specific target and target port.

Cilium nat table information likes below
cilium bpf nat list | grep '[target ip]:[target port]' | grep 'XLATE_SRC' | sort -k7
TCP OUT xxx.xxxx.xxx.xxx:32768 -> [target ip]:[target port] XLATE_SRC [egress ip]:32768 Created=2719514sec HostLocal=0
.
.
.
TCP OUT xxx.xxxx.xxx.xxx:32768 -> [target ip]:[target port] XLATE_SRC [egress ip]:60999 Created=2719514sec HostLocal=0

Thus, we guess the ephemeral ports of a specific egress ip are exhausted, and source ports of egress ip would not be recycled/deleted in nat mapping table after a period of time.
Is there maybe a bug or garbage collect is not working?

Cilium Version

1.11.8

Kernel Version

5.13.0-40-generic

Kubernetes Version

v1.21.14

Sysdump

No response

Relevant log output

No response

Anything else?

Scenario:

How to reproduce?

1. Create a web service in cluster outside.

2. Deploy a pod which can use curl command and apply CiliumEgressNATPolicy through egress ip to web service.

3. Execute bash command in the pod.
   Repeat curl web service until total of running times exceed 28231.
   (Hint: ephemeral port range: net.ipv4.ip_local_port_range = 32768 60999)
   for i in {1..5000}; do [[ $(( $i % 1000 )) -eq 0 ]] && echo $i; curl -sI target ip]:[target port] > /dev/null; done &

4. In cilium pod of egress gateway, you can observe nat mapping variation.

cilium bpf nat list | grep '[target ip]:[target port]' | grep 'XLATE_SRC' | sort -k7
TCP OUT xxx.xxxx.xxx.xxx:32768 -> [target ip]:[target port] XLATE_SRC [egress ip]:32768 Created=2719514sec HostLocal=0
.
.
.
TCP OUT xxx.xxxx.xxx.xxx:32768 -> [target ip]:[target port] XLATE_SRC [egress ip]:60999 Created=2719514sec HostLocal=0`

5. Create another pod and also curl same target and destination port,
   We would receive error msg: "reason No mapping for NAT masquerade" from cilium pod of egress gateway node.
   (e.g. cilium monitor -t drop -vv)

Code of Conduct

• I agree to follow this project's Code of Conduct