/test

@ysksuzuki I'm going to suggest we actually merge in this change.

I've been running the ipsec up/down tests for a bit now. 0/6 so far runs failed:https://github.com/cilium/cilium/actions/runs/14576550343

It does not look like the original issue #37540 is occurring.

@ysksuzuki I'm going to suggest we actually merge in this change.

I've been running the ipsec up/down tests for a bit now. 0/6 so far runs failed:https://github.com/cilium/cilium/actions/runs/14576550343

It does not look like the original issue #37540 is occurring.

Hm I don't follow - isn't this exactly the symptom we're looking for?

(for context - #38757 moved the ipsec configs over into e2e-upgrade. I believe you've been trying to run ipsec-e2e for a repro instead? But that workflow doesn't even do any upgrade/downgrade ...)

Right, this is the issue I saw in #37540. I wasn't aware of that the ipsec configs are moved to e2e-upgrade, sorry.

  🟥 Pod test-conn-disrupt-client-non-backend-node-ipv6-internalip-xlwl5 flow was interrupted (restart count does not match 0 != 1)

Also, ci-ipsec-upgrade fails consitently with this change on v1.17. https://github.com/cilium/cilium/actions/runs/14606495152
I tested on #39097. I think we need to take care of it too.

Wait @julianwiedmann @ysksuzuki

Now I am a bit confused.

I believe you've been trying to run ipsec-e2e for a repro instead? But that workflow doesn't even do any upgrade/downgrade ...)

That's not the case, I've been testing: https://github.com/cilium/cilium/actions/runs/14576550343 like the original issue, which has 0/8 runs failing so far.

Hm I don't follow - isn't this exactly the symptom we're looking for?

That is not the case. The e2e-upgrade tests you point to are failing due to a ctmap flush error which is occurring currently, outside of this PR which turns north-south load balancing tests back on: https://github.com/cilium/cilium/actions/runs/14614535990

That ctmap flush error is being tracked in an entire separate pull request: #39018

Wait @julianwiedmann @ysksuzuki

Now I am a bit confused.

I believe you've been trying to run ipsec-e2e for a repro instead? But that workflow doesn't even do any upgrade/downgrade ...)

That's not the case, I've been testing: https://github.com/cilium/cilium/actions/runs/14576550343 like the original issue, which has 0/8 runs failing so far.

The original issue was for ci-ipsec-upgrade. You're testing ci-ipsec-e2e.

Hm I don't follow - isn't this exactly the symptom we're looking for?

That is not the case. The e2e-upgrade tests you point to are failing due to a ctmap flush error which is occurring currently,

The ctmap error is a red herring. This is what's causing the workflow to fail:

📋 Test Report [cilium-test-1]
❌ 1/1 tests failed (0/0 actions), 120 tests skipped, 0 scenarios skipped:
Test [no-interrupted-connections]:
⛑️ The following owners are responsible for reliability of the testsuite:
- @cilium/sig-datapath (no-interrupted-connections)
- @cilium/ci-structure (.github/workflows/tests-e2e-upgrade.yaml)

And if you scroll up, that's

[-] Scenario [no-interrupted-connections/no-interrupted-connections]
🟥 Pod test-conn-disrupt-client-non-backend-node-ipv6-internalip-xlwl5 flow was interrupted (restart count does not match 0 != 1)

Ahh I was looking at: https://github.com/cilium/cilium/actions/runs/14576551857/job/40950760283

Okay I see it now in ipsec-5. And yeah, just realizing now ipsec upgrade is no more, I mis-linked to the e2e tests looking for ipsec-upgrade.

I am unable to reproduce the issue on my local machine.

This maybe due to the specific version of the kernel ipsec-5 is running on. I'm doing a test now which runs the ipsec-5 configuration with a 6.10 kernel to see if the error is reproduced on this kernel as well.

/test

/test

Interesting, when we change ipsec-5 to use the 6.10 kernel the issue goes away. It's starting to at least look like this is a 5.4 kernel issue specifically.

Just to confirm I'm going to add a few test configs which use the same params which cause the issue but spread over our supported kernel ranges. It should appear to us that only 5.4 fails.

Had a quick look at the sysdump:

• "drop_reason_desc":"UNSUPPORTED_PROTOCOL_FOR_NAT_MASQUERADE","Summary":"ICMPv4 TimeExceeded(TTLExceeded)": addressed by bpf: nat: ICMP v4 improvements #36767, although it won't help with this specific problem. Just reducing the noise.
• "drop_reason_desc":"UNSUPPORTED_PROTOCOL_FOR_NAT_MASQUERADE","Summary":"ICMPv6 TimeExceeded(HopLimitExceeded)": we have upstream support for this ICMP message as well, might also backport.
• "traffic_direction":"INGRESS","file":{"name":"nodeport.h","line":1213},"drop_reason_desc":"FIB_LOOKUP_FAILED": these should be promising, and might also explain why we're only seeing problems on v5.4 (different behavior in fib.h).

/test

This is ready for review.

@julianwiedmann has removed the 5.4 kernel tests which were causing an error here.
Also we have a v1.17 backport PR open which updates the stable workflows to ignore NS tests if kernel version is 5.4 here: #39443

/test