operator: impove handling taints in operator #35785
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
b9770ff to
d18c519
Compare
7215dad to
bdbc7f8
Compare
|
/test |
bdbc7f8 to
cb016c5
Compare
|
/test |
giorio94
approved these changes
Nov 6, 2024
cb016c5 to
b7f7d0f
Compare
|
/test |
squeed
reviewed
Nov 8, 2024
squeed
reviewed
Nov 8, 2024
squeed
reviewed
Nov 8, 2024
squeed
reviewed
Nov 8, 2024
squeed
approved these changes
Nov 11, 2024
b7f7d0f to
80f17ae
Compare
|
/test |
Previously, during each upgrade of Cilium, operator had to taint and untaint nodes whenever Cilium was scheduled and then running. Hovewer, at large clusters, this resulted in multiple parallel requests that were rate-limited on client-go in case of low qps. This resulted in a huge queue of updates pending with stale node information. While simple mitigation of increasing qps works, we should cap max number of requests that operator issues at a given time to reduce staleness of node information. Additionally, it prevents starving other clients of client-go. This commit introduces workers that synchronize taint information instead of relying on unbounded number of controllers. Additionally, we merge node and pod update queues into a single workqueue. Signed-off-by: Marcel Zieba <marcel.zieba@isovalent.com>
80f17ae to
dbc3bab
Compare
|
/test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, during each upgrade of Cilium, operator had to taint and
untaint nodes whenever Cilium was scheduled and then running.
Hovewer, at large clusters, this resulted in multiple parallel requests
that were rate-limited on client-go in case of low qps.
This resulted in a huge queue of updates pending with stale node
information.
While simple mitigation of increasing qps works, we should cap max
number of requests that operator issues at a given time to reduce
staleness of node information. Additionally, it prevents starving other
clients of client-go.
This commit introduces workers that synchronize taint information
instead of relying on unbounded number of controllers.
Additionally, we merge node and pod update queues into a single
workqueue.