service: differentiate UDP and TCP protocols #33434

jibi · 2024-06-27T15:15:33Z

This PR introduces support for protocol differentiation (UDP, TCP) for services.

The high level idea (first commit) is to take into account the L4 protocol specified in a k8s service when populating the bpf lb maps, and then use the packet's protocol to lookup services in those maps.

Next, a new --bpf-lb-proto-diff flag (enabled by default) is introduced. When disabled, the agent strips the protocol from the service object, so even though all the protocol differentiation logic is in place, the control plane sees only services with NONE protocol, effectively falling back to the old behavior. In addition to that also the datapath stops taking into account the protocol when looking up the lb maps.

Next we tackle the reason why the previous attempts didn't make it into the main branch: connection disruptions during upgrades and downgrades. The proposed solution is to keep, during upgrades, existing old style/NONE services as such instead of deleting them and creating new protocol-aware ones. In the control plane before creating a new service, even if protocol differentiation is enabled, we check if the same service with NONE protocol already exists. If that's the case, we don't create a new one. In the datapath we add an extra lookup to the lb maps: if the protocol-aware lookup fails, we set the protocol to 0 (NONE) and try again. This allows to go through upgrades and downgrades without disrupting connections for existing NONE services. This doesn't solve all the cases: if a service has its protocol set and Cilium is downgraded to a version that doesn't support protocol differentiation, such connection will break as the service will be recreated.

Forth commit is just about changing NONE to ANY when dealing with no-proto services.

Finally we update all the unit and ginkgo tests to make them aware of the protocol differentiation logic, as well as the CLI to make a test aware of the protocol differentiation.

I'm currently looking into additional unit tests for controlplane and datapath, as well as additional documentation, but given that the functionality is in this PR should be already good for a first round of reviews.

pkg/loadbalancer/loadbalancer.go

pkg/service/service.go

Documentation/cmdref/cilium-agent.md

daemon/cmd/daemon_main.go

pkg/k8s/watchers/service.go

[ upstream commit 82effe0 ] [ Backporter's notes: Decreased number of expected deletes in TestServiceEventDebounce, as #33434 has not been backported and ANY protocol is not expected. ] When a service has multiple endpoint slices associated with it this will cause Cilium to process each slice separately, causing repeated updating of the service BPF maps and other churn. To reduce the processing in these cases, debounce the processing of the service events by collecting events into a buffer every 200ms. If a new event arrives for a specific service during this duration it is coalesced with the prior one. Signed-off-by: Jussi Maki <jussi@isovalent.com>

[ upstream commit 82effe0 ] [ Backporter's notes: Decreased number of expected deletes in TestServiceEventDebounce, as #33434 has not been backported and ANY protocol is not expected. ] When a service has multiple endpoint slices associated with it this will cause Cilium to process each slice separately, causing repeated updating of the service BPF maps and other churn. To reduce the processing in these cases, debounce the processing of the service events by collecting events into a buffer every 200ms. If a new event arrives for a specific service during this duration it is coalesced with the prior one. Signed-off-by: Jussi Maki <jussi@isovalent.com> Signed-off-by: Fabio Falzoi <fabio.falzoi@isovalent.com>

Reflect cilium#33434 in the BPF tests. Signed-off-by: Julian Wiedmann <jwi@isovalent.com>

Reflect #33434 in the BPF tests. Signed-off-by: Julian Wiedmann <jwi@isovalent.com>

Starting with cilium 1.17 [1][2], it now supports protocol differentiation. Setting the loadBalancer.protocolDifferentiation.enabled will result in ``bpf-lb-proto-diff`` being set in cilium's configmap, which corresponds to this feature. We need this feature in order for the following Conformance test to pass: ``` [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance] [sig-network, Conformance] ``` However, that is not enough for the test to pass. We also need to support `HostPort`s. According to the documentation [3], because we're the standard kube-proxy and not cilium's replacement (``kubeProxyReplacement=false``), we need to also set the Helm chart option ``cni.chainingMode=portmap``. For the test above to pass, we also need to enable [1] cilium/cilium#9207 [2] cilium/cilium#33434 [3] https://github.com/cilium/cilium/blob/v1.17.1/Documentation/installation/cni-chaining-portmap.rst#portmap-hostport

Starting with cilium 1.17 [1][2], it now supports protocol differentiation. Setting the loadBalancer.protocolDifferentiation.enabled will result in ``bpf-lb-proto-diff`` being set in cilium's configmap, which corresponds to this feature. We need this feature in order for the following Conformance test to pass: ``` [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance] [sig-network, Conformance] ``` However, that is not enough for the test to pass. We also need to support `HostPort`s. According to the documentation [3], because we're the standard kube-proxy and not cilium's replacement (``kubeProxyReplacement=false``), we need to also set the Helm chart option ``cni.chainingMode=portmap``. [1] cilium/cilium#9207 [2] cilium/cilium#33434 [3] https://github.com/cilium/cilium/blob/v1.17.1/Documentation/installation/cni-chaining-portmap.rst#portmap-hostport

Starting with cilium 1.17 [1][2], it now supports protocol differentiation. Setting the loadBalancer.protocolDifferentiation.enabled will result in ``bpf-lb-proto-diff`` being set in cilium's configmap, which corresponds to this feature. We need this feature in order for the following Conformance test to pass: ``` [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance] [sig-network, Conformance] ``` However, that is not enough for the test to pass. We also need to support `HostPort`s. According to the documentation [3], because we're the standard kube-proxy and not cilium's replacement (``kubeProxyReplacement=false``), we need to also set the Helm chart option ``cni.chainingMode=portmap``. For the test above to pass, we also need to enable [1] cilium/cilium#9207 [2] cilium/cilium#33434 [3] https://github.com/cilium/cilium/blob/v1.17.1/Documentation/installation/cni-chaining-portmap.rst#portmap-hostport

Starting with cilium 1.17 [1][2], it now supports protocol differentiation. Setting the loadBalancer.protocolDifferentiation.enabled will result in ``bpf-lb-proto-diff`` being set in cilium's configmap, which corresponds to this feature. We need this feature in order for the following Conformance test to pass: ``` [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance] [sig-network, Conformance] ``` However, that is not enough for the test to pass. We also need to support `HostPort`s. According to the documentation [3], because we're the standard kube-proxy and not cilium's replacement (``kubeProxyReplacement=false``), we need to also set the Helm chart option ``cni.chainingMode=portmap``. [1] cilium/cilium#9207 [2] cilium/cilium#33434 [3] https://github.com/cilium/cilium/blob/v1.17.1/Documentation/installation/cni-chaining-portmap.rst#portmap-hostport

* build: adds cilium 1.17.1 rocks (#1086) * Report IPv4 and IPv6 node IPs This is required for cilium 1.17.1 for dualstack scenarios. * Adds cilium 1.17.1 rocks The 1.17.1 cilium rocks have been built, and now we're adding it to k8s-snap. Note that this version requires the nodes (kubelet) to also report their IPv4 and IPv6 addresses for dualstack scenarios. k8sd should be adding both of them to kubelet's --node-ip argument. Updates gateway-api chart to 1.2.0. * cilium: Enables cilium session affinity (#1146) Session affinity is disabled by default in the cilium helm chart. Note that there are a few session affinity related Conformance tests which are failing because of this fact: ``` [sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance] [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance] [sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [Conformance] [sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Conformance] ``` Enabling this should resolve the failures for the tests mentioned above. Note that the ``sessionAffinity`` helm chart option was introduced in 1.12.0. * feat: Enables cilium protocol differentiation (#1170) Starting with cilium 1.17 [1][2], it now supports protocol differentiation. Setting the loadBalancer.protocolDifferentiation.enabled will result in ``bpf-lb-proto-diff`` being set in cilium's configmap, which corresponds to this feature. We need this feature in order for the following Conformance test to pass: ``` [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance] [sig-network, Conformance] ``` However, that is not enough for the test to pass. We also need to support `HostPort`s. According to the documentation [3], because we're the standard kube-proxy and not cilium's replacement (``kubeProxyReplacement=false``), we need to also set the Helm chart option ``cni.chainingMode=portmap``. [1] cilium/cilium#9207 [2] cilium/cilium#33434 [3] https://github.com/cilium/cilium/blob/v1.17.1/Documentation/installation/cni-chaining-portmap.rst#portmap-hostport * feat: Update the cilium rock versions (#1325) We have recently reduced the image size of the cilium rock from 1.47 GB to 811 MB. This should reduce the amount of time spent by the Kubernetes nodes to become Ready as well. * use main branch in rock repo --------- Co-authored-by: Claudiu Belu <claudiu.belu@canonical.com>

julianwiedmann · 2025-04-30T05:29:58Z

Can we already discuss what a deprecation schedule for this flag would look like? :)

we could deprecate it in v1.18 and remove it completely in 1.19 🤔 different story for the backward compatibility logic for services with protocol=ANY, that could potentially stay there for a long time (as we have no way to control whether users have recreated their services/there are no more protocol=ANY services around)

Tracking this deprecation here: #39249

Starting with cilium 1.17 [1][2], it now supports protocol differentiation. Setting the loadBalancer.protocolDifferentiation.enabled will result in ``bpf-lb-proto-diff`` being set in cilium's configmap, which corresponds to this feature. We need this feature in order for the following Conformance test to pass: ``` [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance] [sig-network, Conformance] ``` However, that is not enough for the test to pass. We also need to support `HostPort`s. According to the documentation [3], because we're the standard kube-proxy and not cilium's replacement (``kubeProxyReplacement=false``), we need to also set the Helm chart option ``cni.chainingMode=portmap``. [1] cilium/cilium#9207 [2] cilium/cilium#33434 [3] https://github.com/cilium/cilium/blob/v1.17.1/Documentation/installation/cni-chaining-portmap.rst#portmap-hostport

* feat: allow enabling cilium SCTP (#1088) We'll allow enabling the Cilium SCTP feature (beta) through an annotation. https://docs.cilium.io/en/latest/configuration/sctp/ * cilium: Enables cilium session affinity (#1146) Session affinity is disabled by default in the cilium helm chart. Note that there are a few session affinity related Conformance tests which are failing because of this fact: ``` [sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance] [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance] [sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [Conformance] [sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Conformance] ``` Enabling this should resolve the failures for the tests mentioned above. Note that the ``sessionAffinity`` helm chart option was introduced in 1.12.0. * feat: Enables cilium protocol differentiation (#1170) Starting with cilium 1.17 [1][2], it now supports protocol differentiation. Setting the loadBalancer.protocolDifferentiation.enabled will result in ``bpf-lb-proto-diff`` being set in cilium's configmap, which corresponds to this feature. We need this feature in order for the following Conformance test to pass: ``` [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance] [sig-network, Conformance] ``` However, that is not enough for the test to pass. We also need to support `HostPort`s. According to the documentation [3], because we're the standard kube-proxy and not cilium's replacement (``kubeProxyReplacement=false``), we need to also set the Helm chart option ``cni.chainingMode=portmap``. [1] cilium/cilium#9207 [2] cilium/cilium#33434 [3] https://github.com/cilium/cilium/blob/v1.17.1/Documentation/installation/cni-chaining-portmap.rst#portmap-hostport * fix: Minor differences Signed-off-by: Homayoon (Hue) Alimohammadi <homayoon.alimohammadi@canonical.com> --------- Signed-off-by: Homayoon (Hue) Alimohammadi <homayoon.alimohammadi@canonical.com> Co-authored-by: Lucian Petrut <lpetrut@cloudbasesolutions.com> Co-authored-by: Claudiu Belu <claudiu.belu@canonical.com>

jibi linked an issue Jun 27, 2024 that may be closed by this pull request

Differentiate between UDP and TCP services #9207

Closed

jibi force-pushed the pr/jibi/differentiate-udp-tcp-svcs-take-4 branch 5 times, most recently from 8af0128 to 533ac80 Compare July 3, 2024 10:08

jibi mentioned this pull request Jul 12, 2024

Differentiate between UDP and TCP services #9207

Closed