When upgrading to 1.11.15, 1.12.8, or 1.13.1 with IPsec enabled, cross-node connectivity fails completely. New cluster installations on those versions are okay. This issue happens because stale Linux XFRM states and policies are left behind and they conflict with the new states and policies we install on those versions.

Mitigation

The best is obviously to wait for the fix before upgrading. If that is not possible or if the migration already happened, there are three solutions:

1. Restart or replace the nodes such that new nodes with a clean state are used.
2. Manually clean the XFRM states and policies on all nodes with ip x s flush && ip x p flush from Cilium agent pods.
3. Disable IPsec.