-
Notifications
You must be signed in to change notification settings - Fork 3.4k
ipsec: Clean up stale XFRM policies and states #24773
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ipsec: Clean up stale XFRM policies and states #24773
Conversation
fe461b9
to
21c438e
Compare
These wildcard variables will be used by a later commit in the IPsec logic. Signed-off-by: Paul Chaignon <paul@cilium.io>
21c438e
to
b8f97c1
Compare
UpsertIPsecEndpoint is currently unable to replace stale XFRM states. We use XfrmStateAdd, which fails with EEXIST if a state with the same key (IPs, SPI, and mark) already exists. We can't use XfrmStateUpdate because it fails with ESRCH is no state with the specified key exist. Note we don't have the same issue for XFRM policies because XfrmPolicyUpdate doesn't return ESRCH if no such policy already exists. No idea why the two APIs are not consistent. We therefore need to implement a proper 'update or insert' logic for XFRM states ourselves. To that end, we first check if the state we want to add already exists. If it doesn't, we attempt to add it. If it fails with EEXIST, we know that some other state is conflicting. In that case, we attempt to remove any conflicting XFRM states that are found and then attempt to add the new state again. To find conflicting XFRM states, we use the same logic as the kernel does (cf. __xfrm_state_lookup). Signed-off-by: Paul Chaignon <paul@cilium.io>
This commit adds a catch-all XFRM policy for outgoing traffic that has the encryption bit. The goal here is to catch any traffic that may passthrough our encryption while we are replacing XFRM policies & states. Those operations cannot always be performed atomically so we may have brief moments where there is no XFRM policy to encrypt a subset of traffic. This policy ensures we drop such traffic and don't let it flow in plain text. We do need to match on the mark because there is also traffic flowing through XFRM that we don't want to encrypt (e.g., hostns traffic). Signed-off-by: Paul Chaignon <paul@cilium.io>
We recently changed our XFRM states and policies (IPs and marks). We however failed to remove the stale XFRM states and policies and it turns out that they conflict (e.g., the kernel ends up picking the stale policies for encryption instead of the new one). This commit therefore cleans up those stale XFRM states and policies. We can identify them based on mark values and masks (we switched from 0xFF00 to 0XFFFFFF00). The new XFRM states and policies are added as we receive the information on remote nodes. By removing the stale states and policies before the new ones are installed for all nodes, we could cause plain-text traffic on egress and packet drops on ingress. To ensure we never let plain-text traffic out, we will clean up the stale config only once the catch-all default-drop policy is installed. In that way, if there is a brief moment where, for a connection nodeA -> nodeB, we don't have a policy, traffic will be dropped instead of sent in plain-text. For each connection nodeA -> nodeB, those packet drops on egress and ingress of nodeA will happen between the time we replace the BPF datapath and the time we've installed the new XFRM state and policy corresponding to nodeB. Waiting longer to remove the stale states and policies doesn't impact the drops as they will keep happening until the new states and policies are installed. This is all happening on agent startup, as soon as we have the necessary information from k8s. Signed-off-by: Paul Chaignon <paul@cilium.io>
b8f97c1
to
5562165
Compare
/test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm for cli team files
should this be a release blocker for april patch release (1.13.2, 1.12.9, 1.11.16)? |
oh i guess it already is, i see the corresponding issue has release-blocker labels: #24780 |
IP: wildcardIPv4, | ||
Mask: net.IPv4Mask(0, 0, 0, 0), | ||
} | ||
wildcardIPv6 = net.ParseIP("0::0") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: This can be ::
.
func removeStaleStates(family int) { | ||
states, err := netlink.XfrmStateList(family) | ||
if err != nil { | ||
log.WithError(err).Error("Cannot get XFRM states") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we need return here?
func removeStalePolicies(family int) { | ||
policies, err := netlink.XfrmPolicyList(family) | ||
if err != nil { | ||
log.WithError(err).Error("Cannot get XFRM policies") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we need return here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, good point. I'll do that as a follow up, with a bunch of other IPsec code improvements, in the interest of merging this quickly and unblocking backports.
I've checked the source for XfrmPolicyList
and, if there is an error, an empty slice is always return, so the subsequent for loop will just be a noop.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM with non-blocking nits.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cli
owned files (the logger) LGTM. I'll defer to others for now on the state replacement correctness.
ConformanceEKS failed with the recently fixed know breakage #24774. Other tests are passing. Reviews are in. Marking ready to merge so I can get started on backports sooner rather than later. |
First commit is a bit of refactoring. Second implements a proper function to update XFRM states. Third adds a catch-all default-drop policy to avoid leaking plain-text traffic during the cleanup. Fourth commit implements the cleanup itself.
See commit descriptions for details.
Fixes: #24030.
Fixes: #24780.
Updates: #24010.