Skip to content

network problem caused by absent reverse entry in snat_v4_external map  #27821

@ghost

Description

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

One application in our prod environment encounters a network problem. When this application accesses one specific port of a remote host, the pod cannot get the response, the node get reponse but send RST. After restart the application, the problem still occur on another port. We find that in cilium_snat_v4_external map, there should be a pair of entries (tuples and states), but the reverse one is absent. After manually adding the lost entry, the network problem is fixed. The tuples in ct4 map exist.

cilium monitor -t drop shows no event. I think the eBPF program return NAT_PUNT_TO_STACK, which is ignored.

Cilium is configured as follows:

  • ebpf masquerade enabled
  • ebpf host routing enabled
  • strict kube-proxy replacement
  • native routing (tunnel: false)
  • cluster-pool mode IPAM
  • DSR is not used

eBPF program seems adding and deleting snat ingress and egress entries at same time, deleteSwappedMapping4() in conntrack GC seems delete one entry, but the comment says that this is used for DSR.

I don't know how to reproduce the problem, and am not sure this is a bug. Is there a situation that only one of the entries is deleted without DSR? Thanks.

Cilium Version

1.11.17

Kernel Version

5.10

Kubernetes Version

1.24.3

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.need-more-infoMore information is required to further debug or fix the issue.staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions