-
Notifications
You must be signed in to change notification settings - Fork 3.4k
operator: Improve identity GC efficiency #17359
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
operator: Improve identity GC efficiency #17359
Conversation
341f25b
to
69e7ce8
Compare
test-me-please Job 'Cilium-PR-K8s-1.19-kernel-5.4' failed and has not been observed before, so may be related to your PR: Click to show.Test Name
Failure Output
If it is a flake, comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just one minor nit to make the warning a little bit more clear.
Noting also what we found on friday, which was that etcdconfig.etcd.qps can be a limiting factor here.
This is useful in warning or error level messages to help nudge the user in the right direction when troubleshooting. Signed-off-by: Chris Tarazi <chris@isovalent.com>
69e7ce8
to
728b2b9
Compare
test-me-please Edit: Images timed out |
test-me-please |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, only a small nit.
728b2b9
to
3cd71e1
Compare
With this commit, the identity GC rate limit (--identity-gc-rate-interval) becomes the effective rate at which identities are garbage collected. Previously, the identity GC interval (--identity-gc-interval) would cause the Operator to GC for that much time, then the sleep for that much time, rinse and repeat, effectively halving the rate. To use concrete numbers for an example, let's say our interval is 5m and our GC rate interval is 1000 per minute. It would mean that previously, we would GC 5000 identities at a maximum for 10m (assuming that deletion takes 0s). How was that calculated? Each minute, we GC 1000 identities. After 5m, we have GC'd 5000 identities. But now we have to sleep for 5m because that's our GC interval. Hence making our effective GC rate 500 per minute (instead of being 1000/m). Now, we compute the time taken to perform the actual GC and subtract that from the interval. So in our above example, we eliminate the dead time of 5m and avoid slashing our effective GC rate in half. This change allows the Operator to keep up with the demand more efficiently. The Operator will warn if the GC duration took longer than the interval and set the sleep duration to 0. Suggested-by: Joe Stringer <joe@cilium.io> Suggested-by: Dan Wendlandt <dan@isovalent.com> Signed-off-by: Chris Tarazi <chris@isovalent.com>
3cd71e1
to
d4fad43
Compare
With this commit, the identity GC rate limit
(--identity-gc-rate-interval) becomes the effective rate at which
identities are garbage collected. Previously, the identity GC interval
(--identity-gc-interval) would cause the Operator to GC for that much
time, then the sleep for that much time, rinse and repeat, effectively
halving the rate.
To use concrete numbers for an example, let's say our interval is 5m and
our GC rate interval is 1000 per minute. It would mean that previously,
we would GC 5000 identities at a maximum for 10m (assuming that deletion
takes 0s). How was that calculated? Each minute, we GC 1000 identities.
After 5m, we have GC'd 5000 identities. But now we have to sleep for 5m
because that's our GC interval. Hence making our effective GC rate 500
per minute (instead of being 1000/m).
Now, we compute the time taken to perform the actual GC and subtract
that from the interval. So in our above example, we eliminate the dead
time of 5m and avoid slashing our effective GC rate in half. This change
allows the Operator to keep up with the demand more efficiently. The
Operator will warn if the GC duration took longer than the interval and
set the sleep duration to 0.
Suggested-by: Joe Stringer joe@cilium.io
Suggested-by: Dan Wendlandt dan@isovalent.com
Signed-off-by: Chris Tarazi chris@isovalent.com