Summary

Introduce a new implementation of the proxy redirect handling in the eBPF datapath using a combination of socket lookups and socket assign (+ packet redirects where necessary to ensure that socket assign occurs at TC ingress).

This functionality is disabled by default in the daemon, but enabled in new v1.9 deployments via Helm. When enabled on Linux 5.7 or later, it removes use of the -j TPROXY target, and removes Cilium's usage of the upper 16 bits of the mark for the purpose of sharing the tproxy target port with iptables rules.

API impact

Three new monitor error codes are added for cases where the redirect cannot be applied, meaning that traffic subject to L7 policy will be dropped if there is a configuration or lookup issue:

• DROP_PROXY_LOOKUP_FAILED
  • Lookup for the target proxy socket failed. This will trigger drop when the datapath cannot find any establish proxy connection for this tuple AND the datapath cannot find a TPROXY listening socket corresponding to the redirect port in the eBPF policy map. If the proxy is not available for any reason then this could be a reason to see this error.
• DROP_PROXY_SET_FAILED
  • Failed to assign the socket to the packet context. Typically programmer error but may trigger if the socket which was found previously is unavailable or has incompatible options set (eg SO_REUSEPORT). In the event of this error, the default monitor logs will additionally provide an errno to determine which subfailure occurred.
• DROP_PROXY_UNKNOWN_PROTO
  • Safety catchall in case we attempt to redirect traffic other than TCP/UDP to a socket. As of Linux 5.7, no other L4 protocols are supported for tproxy redirection.

Implementation

Previously, in the eBPF datapath during policy apply and redirect processing, we have relied purely on the tproxy port which is available in the policymap to prepare the packet for TPROXY redirection. Mechanically this path involved performing a policy lookup using the src/dst identity, then in the policymap entry, using non-zero redirect port as a signal for redirect. This port would then be set into the ctx->mark along with some bits to identify this traffic as "needs proxy redirection". Then the packet would be passed up the stack towards an iptables PREROUTING rule which would match on the mark then perform a socket lookup and assign the socket to the skb.

The first step of the new implementation is to ensure that the relevant forwarding logic has access to the full packet tuple. There is one fairly significant condition for applying the socket to the skb: It must be done at TC ingress. In many cases, Cilium already applies all logic at TC ingress eBPF programs so this isn't too difficult. For endpoint-routes mode however we apply pod ingress policy at the TC egress hook point, and therefore must first hairpin to another device, then extract the tuple & perform the socket lookups to assign the socket to the skb.

With this in place, the actual lookup & assign is relatively straightforward. First we must look for any established connections corresponding to the tuple in the local network namespace.These represent existing proxy connections that were already established. If we find the socket here, we assign it to the skb & pass up the stack. If we don't find it, then we zero out the tuple and set the TPROXY redirect port from the policy into the tuple and perform a second lookup for listen sockets. This should provide the parent listener socket associated with the proxy which we can assign to establish a new connection in the proxy.

For now, this PR introduces dedicated tuple extraction logic in bpf_host.c for the endpoint routes mode case rather than sharing it with host firewall, primarily because it predates host firewall but also because the tuple extraction needs to be performed if eBPF tproxy is enabled but host firewall is not. A future cleanup could look at shifting tuple extraction into common code which would be enabled by either path and shared. Further optimizations could be explored to perhaps avoid further packet touching in this prog by sharing some of that metadata from the previous program that was run at TC egress of the endpoint subject to L7 policy.

This PR replaces the per-redirect iptables rule logic but not the static -m socket logic as this requires further exploration to determine the simplest path forward. Fortunately, the newly added connectivity checks for proxy paths makes it easy to reproduce failures on these paths to allow investigation and experimentation.

Testing

The recently-added examples/kubernetes/connectivity-check-proxy.yaml is now added to the main CI for testing various proxy paths to prevent regressions. Additionally I have manually deployed this YAML for debugging in a GKE environment (default instructions, endpoint-routes mode).

The BPF checks / checkpatch (pull_request) GitHub action job doesn't seem to like my use of macros to avoid duplicating IPv4/IPv6 implementations, but I've looked through the complaints and AFAICT they're all safe to ignore.

Tasks

• Daemon logic to override enable/disable
  • Iptables compatibility mode that still configures rules for compatibility with existing rulesets (eg kube-proxy)
• IPv6 support
• Endpoint-routes mode validation
• CI tests to validate the new paths
• Issue description for reviewers :-)

Deferred to follow up PRs

• Replace -m socket rules - I looked briefly into this, should be doable in one of a couple of ways but it's more involved so I figured it's better to get the basics in first.
• This PR touches the issue Unable to upgrade from Cilium 1.8 to 1.9 without breaking compatibility #12288 but does not resolve it.

I marked this 'release-note/major' with the (ambitious?) goal of it being listed with other PRs as part of iptables removal for v1.9, but ideally it should have no user-facing impact :-) Happy to move it to /minor if we think this is not v1.9 release higlight worthy.

Related: #9921