Skip to content

If pods establish more external connections than the value specified by --bpf-nat-global-max, it causes a DROP in SNAT behavior for cil-to-netdev when using iptables masquerade #36572

New issue
New issue
Closed
Closed
If pods establish more external connections than the value specified by --bpf-nat-global-max, it causes a DROP in SNAT behavior for cil-to-netdev when using iptables masquerade#36572
Labels
area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.feature/snatRelates to SNAT or Masquerading of traffickind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
@gyutaeb

Description

@gyutaeb
gyutaeb
opened on Dec 13, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Version

equal or higher than v1.16.4 and lower than v1.17.0

What happened?

When using KubeProxyReplacement: Strict with Masquerading: IPTables [IPv4: Enabled], the source address is already nat-ed by iptables. However, cil-to-netdev performs an additional unnecessary SNAT behavior. It also unnecessarily occupies entry in the SNAT_MAPPING map. And if it exceeds SNAT_COLLISION_RETRIES, it leads to connection failures.

cilium/bpf/lib/nat.h

Lines 232 to 243 in daea10f

if (__snat_create(map, &rtuple, &rstate) == 0)
goto create_nat_entry;
port = __snat_clamp_port_range(target->min_port,
target->max_port,
retries ? port + 1 :
(__u16)get_prandom_u32());
}
/* Loop completed without finding a free port: */
ret = DROP_NAT_NO_MAPPING;
goto out;

How can we reproduce the issue?

To easily reproduce the issue, you can reduce --bpf-nat-global-max to 1024 and send multiple connection from a pod to external packets. This should trigger the issue immediately.

fortio load -qps 1000 -c 1000 -t 100s -allow-initial-erros -keepalive=false http://{EXTERNAL_IP}

Cilium Version

cilium-cli: v0.16.15 compiled with go1.22.5 on darwin/arm64
cilium image (default): v1.16.0
cilium image (stable): v1.16.4
cilium image (running): v1.16.4

Kernel Version

Linux gyu-test-1 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.26.4

Regression

No response

Sysdump

No response

Relevant log output



Anything else?


No response


Cilium Users Document


    

  •  Are you a user of Cilium? Please add yourself to the Users doc
    • 



Code of Conduct


    

  •  I agree to follow this project's Code of Conduct
    • 

Metadata
Metadata
Assignees
No one assigned
    Labels
    area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.feature/snatRelates to SNAT or Masquerading of traffickind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
    Type
    No type
    Projects
    No projects
    Milestone
    No milestone
    Relationships
    None yet
    Development
    No branches or pull requests
    Issue actions