/test

Hello @aditighag,

Thanks for running the tests. I've reviewed the failures:

"The action 'Wait for images to be available (downgrade)' has timed out after 10 minutes."

• This seems to be a build error unrelated to my PR.

"Failed to execute tcpdump on cilium-test-3/host-netns-wtqnd (kind-worker2): command failed (pod=cilium-test-3/host-netns-wtqnd, container=): error reading from error stream: next reader: websocket: close 1006 (abnormal closure): unexpected EOF"

• This appears to be an error during the test setup. Tcpdump needs to run to check if the captured packets match some condition.

"command 'curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code}\n --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[fc00:c111::5]:30755' failed: command failed (pod=cilium-test-4/host-netns-non-cilium-pjg8p, container=host-netns-non-cilium): command terminated with exit code 28"

• This might indicate my PR is breaking TCP connections.

"Expected to capture packets, but none found. This check might be broken."

• This could be a flaky test.

"timeout=5s error='rpc error: code = Unavailable desc = dns: A record lookup error: lookup hubble-peer.kube-system.svc.cluster.local. on 10.100.0.10:53: read udp 192.168.122.27:33793->10.100.0.10:53: i/o timeout' subsys=hubble-relay"

• This might be due to my patch overclosing sockets.

Could you help me distinguish between CI issues and those caused by my fix?

Additionally, I'd appreciate your feedback on the code itself. It works in my environment without issues, but I'm open to reworking my PR if you see a better solution.

Best regards.

Hello @aditighag,

Thank for fixing the release note, I thought my PR title was suitable as a release note.

I made the following test:

• deploy Cilium 1.17.0
• set bpf-lb-proto-diff to false in the Cilium configmap
• restart the Cilium daemonset
• restart the daemonset of the service that is receiving packets from long lived UDP connections

After looking at Hubble, I still see the symptomatic drops of UDP packets going to non-existing backends.

The PR was rebased and I accepted your change.

Best Regards.

/test

@foyerunix All tests passed. Can you drop my sign-off from the commit? Once you've force pushed to the PR, we need not run the full CI test suite before merging the PR.

@foyerunix I wasn't able to reproduce the issue locally. See snippets from my testing below.

Can you share the repro steps including your service manifest?

$ k get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
echo            ClusterIP   10.96.132.231   <none>        80/TCP,69/UDP     119s

$ ks logs cilium-mtskr | grep -i handling
time="2025-03-04T21:31:24.278942184Z" level=info msg="handling udp connections to deleted backend 10.244.1.54:80/TCP" subsys=service
time="2025-03-04T21:31:24.279089498Z" level=info msg="handling udp connections to deleted backend 10.244.1.54:69/UDP" subsys=service

diff --git a/pkg/service/connections.go b/pkg/service/connections.go
index 292d1ba4db..b12465a541 100644
--- a/pkg/service/connections.go
+++ b/pkg/service/connections.go
@@ -25,6 +25,7 @@ import (
 var opSupported = true

 func (s *Service) TerminateUDPConnectionsToBackend(l3n4Addr *lb.L3n4Addr) {
+       log.Infof("handling udp connections to deleted backend %v", l3n4Addr)
        // With socket-lb, existing client applications can continue to connect to
        // deleted backends. Destroy any client sockets connected to the deleted backend.
        if !(option.Config.EnableSocketLB || option.Config.BPFSocketLBHostnsOnly) {

Hello @aditighag,

I dropped your sign-off from the commit.

I didn't build a minimal reproducer that allow to reproduce the issue without having to install the Datadog agent.

I have looked at your snippets and:

time="2025-03-04T21:31:24.278942184Z" level=info msg="handling udp connections to deleted backend 10.244.1.54:80/TCP" subsys=service
time="2025-03-04T21:31:24.279089498Z" level=info msg="handling udp connections to deleted backend 10.244.1.54:69/UDP" subsys=service

Is not what I got when I added debugging. You should have: 10.244.1.54:69/ANY.

Here is my service spec:

spec:
  clusterIP: [redacted]
  clusterIPs:
  - [redacted]
  internalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: dogstatsdport
    port: 8125
    protocol: UDP
    targetPort: 8125
  - name: traceport
    port: 8126
    protocol: TCP
    targetPort: 8126
  selector:
    app: datadog
  sessionAffinity: None
  type: ClusterIP

Best Regards.

What cilium version have you installed on your cluster? Can you try installing the latest version?

Seems like this stalled - are we looking for further confirmation from @foyerunix or do we think the change is right? As far as I follow it seems like the calling code doesn't discriminate on the protocol of the service, and ANY could mean there are UDP connections. Or do we already expand out ANY to each individual protocol in l3n4Addr?

I couldn't immediately find tests for this functionality but I guess if we extended the testsuite that could help to prove the correct behavior.

Hello @joestringer,

I saw that @aditighag asked me to do a test with the latest Cilium version. Meanwhile I don't have the Datadog agent installed in a sandbox environment. I need either to start a Cilium upgrade cycle on my clusters, or to take the risk to just launch the latest Cilium agent on an impacted cluster as is. I'm still unsure about what to do.

I run Cilium 1.17.0.

Best Regards.

Hi @foyerunix We are getting more test coverage as part of this PR - #37600. /cc @tommyp1ckles
@tommyp1ckles Would you recommend @foyerunix to cherry pick your commit, and see if the issue he reported still persists?

Hello @aditighag,

I can confirm that I have the same issue with Cilium 1.17.2.

I have looked at your snippets and:

time="2025-03-04T21:31:24.278942184Z" level=info msg="handling udp connections to deleted backend 10.244.1.54:80/TCP" subsys=service
time="2025-03-04T21:31:24.279089498Z" level=info msg="handling udp connections to deleted backend 10.244.1.54:69/UDP" subsys=service

Does this mean that we also attempt to close TCP from above, but eventually fail since socket cookie + TCP doesn't exist?
I think temporary we should land this, but iirc @tommyp1ckles might be following up with actually terminating TCP as well which should then also land till 1.18.

/test

Hi @foyerunix Could you rebase your PR?

Hi @aditighag,

Done.

Best Regards.

/test

🟥 failed to flush ct entries: %w command failed (pod=kube-system/cilium-zds6t, container=cilium-agent): "time="2025-04-16T15:13:14.440649371Z" level=info msg="completed ctmap gc pass" aliveEntries=0 deleted=3611 family=ipv4 proto=TCP skipped=0 subsys=map-ct\ntime="2025-04-16T15:13:14.444586669Z" level=info msg="completed ctmap gc pass" aliveEntries=0 deleted=422 family=ipv4 proto=non-TCP skipped=0 subsys=map-ct\ntime="2025-04-16T15:13:14.446563167Z" level=info msg="completed ctmap gc pass" aliveEntries=0 deleted=120 family=ipv6 proto=TCP skipped=0 subsys=map-ct\ntime="2025-04-16T15:13:14.447961547Z" level=info msg="completed ctmap gc pass" aliveEntries=0 deleted=66 family=ipv6 proto=non-TCP skipped=0 subsys=map-ct\n"

Hit unrelated CI issue.

Context - #38956.

Hey @foyerunix, thanks for the fix!

We just ran into this very issue!

Could you help me understand the new behavior?

Once s.backendConnectionHandler.Destroy(...) is called, what is going to happen, from the perspective of a client attempting writes through a connected udp socket? Do such clients get to continue using the existing socket fd or do they need to make a new one?

Hello @deadok22,

My PR doesn't change the behavior of socketLB socket termination. It just ensure that UDP connections are terminated when the LB type is "ANY". As I understand, the client need to open a new socket as the kernel forcefully close the existing one.

👋 @foyerunix @aditighag curious, is this essentially caused by the termination code assuming that all services are created with L4 protocol differentation (-> #33434) - and we didn't consider services that were created on a pre-v1.17 cluster with ANY protocol?

If so I think we should backport this fix to v1.17. And add some tests to confirm that we're handling such legacy service definitions as expected.

Hello @julianwiedmann,

I can confirm that my Datadog service was created before upgrading to Cilium 1.17.

When loadBalancer.protocolDifferentiation.enabled=false, the service protocol is treated as ANY. In this case, TerminateUDPConnectionsToBackend does not terminate UDP sockets. Since v1.17 supports protocol differentiation but it may be disabled, we should backport this fix.