You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We sometimes scale down unused control planes to preserve resources. They can be scaled up on demand, so it's always meant as a transitory state.
We observed some issues in our cluster in the past. There were of course the network errors that the controller-runtime watches logged because the control plane was unavailable, but there also was a strange behavior where new control planes would not get valid certificates.
I decided to do the full cleanup with finalizer when the TCP is scaled to zero, because this mirrors what we see in our environments. Sometimes control planes are just deleted by users after they notice they were scaled down, i.e. were unused for a while.
Preliminary testing of this change in our environment looked promising, but I'll let it sit over the weekend to get more data.
Another issue we see when scaling TCPs down to 0 is that the datastore controllers keeps logging channel is full well after the control planes are scaled back up again.
I have yet to figure out 1) why it fills up in the first place and 2) why it stays full after the scale up.
Another issue we see when scaling TCPs down to 0 is that the datastore controllers keeps logging channel is full well after the control planes are scaled back up again
I suspect it's related to our misbehaviour with controllers: we're pushing a generic event to the source channel, which pushes to the controllers unable to run.
Are you still facing this error despite the proposed changes?
Another issue we see when scaling TCPs down to 0 is that the datastore controllers keeps logging channel is full well after the control planes are scaled back up again
I suspect it's related to our misbehaviour with controllers: we're pushing a generic event to the source channel, which pushes to the controllers unable to run.
Are you still facing this error despite the proposed changes?
Yes, and it looks like it's logged for all TCPs (even the ones that are scaled up) when a TCP event is received, e.g. by creating, updating or deleting a TCP.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We sometimes scale down unused control planes to preserve resources. They can be scaled up on demand, so it's always meant as a transitory state.
We observed some issues in our cluster in the past. There were of course the network errors that the controller-runtime watches logged because the control plane was unavailable, but there also was a strange behavior where new control planes would not get valid certificates.
I decided to do the full cleanup with finalizer when the TCP is scaled to zero, because this mirrors what we see in our environments. Sometimes control planes are just deleted by users after they notice they were scaled down, i.e. were unused for a while.
Preliminary testing of this change in our environment looked promising, but I'll let it sit over the weekend to get more data.
Another issue we see when scaling TCPs down to 0 is that the datastore controllers keeps logging
channel is fullwell after the control planes are scaled back up again.I have yet to figure out 1) why it fills up in the first place and 2) why it stays full after the scale up.