EndpointManager: fix deadlock when releasing an endpoint #21771
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
christarazi
reviewed
Oct 17, 2022
There was a problem hiding this comment.
Nice find! It seems that dae07b5 introduced this so adding Fixes: ... to the commit would be good.
162d562 to
e023308
Compare
michi-covalent
approved these changes
Oct 17, 2022
michi-covalent
approved these changes
Oct 17, 2022
michi-covalent
approved these changes
Oct 17, 2022
In high-churn clusters, there can be a three-party deadlock between the EndpointManager, the PolicyRepository, and a given Endpoint. One of the "links in the chain" is merely trying to get the container ID and namespace+name of an Endpoint for logging. Which we already have. So, rather than trying to lock an Endpoint to get it's identifiers again, just use the copy we already have. Fixes: dae07b5 (endpointmanager: Remove goroutine for ID release) Signed-off-by: Casey Callendrello <cdc@isovalent.com>
e023308 to
1e37eb3
Compare
christarazi
approved these changes
Oct 17, 2022
|
/test |
|
/test |
|
/test |
|
Fairly certain the test failures are a flake here. Test-runtime failed with and k8s-1.16-kernel-4.9 fails two of the connectivity tests. Given that this just changes an error log message, they're not related. |
Yep, that's #21735. |
aanm
approved these changes
Oct 19, 2022
|
Nice work! In future can you put the relevant stack traces into the commit message? This can help others to review and learn about this fix, and also makes it easier to search for these stack traces in future in case they hit this issue independently. Then we can compare another failure vs. this fix and confirm whether this fixes the deadlock or not. |
This was referenced Nov 15, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In high-churn clusters, there can be a three-party deadlock between the EndpointManager, the PolicyRepository, and a given Endpoint. One of the "links in the chain" is merely trying to get the container ID and namespace+name of an Endpoint for logging. Which we already have.
So, rather than trying to lock an Endpoint to get it's identifiers again, just use the copy we already have.
Fixes: dae07b5 (endpointmanager: Remove goroutine for ID release)
Signed-off-by: Casey Callendrello cdc@isovalent.com