feat: enhancing concurrent reconciliations #790
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for kamaji-documentation ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
e2e is successful, it's just a false positive, happened time to time with GH Actions. |
|
I got 5/10 with |
|
Why those TCP are in It seems to me it's unrelated to the changes introduced with this PR. Is the Deployment and related Pods matching the expected replicas? |
c8d297f to
5c65a06
Compare
|
@avorima it seems we can skip checking the channel status, I just added a safety net with a timeout for the Generic Event communication. It would be great if you could give it a shot. |
Channels used for GenericEvent feeding for cross controllers triggers are now buffered according to the --max-concurrent-tcp-reconciles: this is required to avoid channel full errors when dealing with large management clusters serving a sizeable amount of Tenant Control Planes. Increasing this value will put more pressure on memory (mostly for GC) and CPU (provisioning multiple certificates at the same time). Signed-off-by: Dario Tranchitella <dario@tranchitella.eu>
Signed-off-by: Dario Tranchitella <dario@tranchitella.eu>
Signed-off-by: Dario Tranchitella <dario@tranchitella.eu>
Signed-off-by: Dario Tranchitella <dario@tranchitella.eu>
This change introduces a grace period of 10 seconds before abruptly terminating the Tenant Control Plane deployment, allowing the soot manager to complete its exit procedure and avoid false positive errors due to API Server being unresponsive due to user deletion. Aim of this change is reducing the amount of false positive errors upon mass deletion of Tenant COntrol Plane objects. Signed-off-by: Dario Tranchitella <dario@tranchitella.eu>
5c65a06 to
1f0d3bd
Compare
WatchesRawSource is non blocking, no need to check if channel is full. To prevent deadlocks a WithTimeout check has been introduced. Signed-off-by: Dario Tranchitella <dario@tranchitella.eu>
1f0d3bd to
5ec7ebc
Compare
Did you increase the CPU limits? That caused all kinds of weird issues for me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Strictly related to #787, started from a benchmark of Kamaji in reconciling multiple Tenant Control Planes.
When dealing with multiple Tenant Control Plane creations, my suggestion is to increase the
--max-concurrent-tcp-reconcilesflag according to the benchmarked scenarios.e.g.: default value of
1and creating dozen of clusters:When increasing the said flag (e.g.: 10)