LB-IPAM: Fix performance fixes #39278
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
During scale testing where only services were being added, the CPU profile showed that a large amount of time was spent revalidating already allocated IPs in pools. This was unexpected. It turns out that when we patch the pool to update the status with new counts, we also observe our own change. Our pool upsert logic for revalidation is called, assuming the spec must have changed. But most of the time its just the status that changed. This commit adds a deepequal check on the pool spec so we only do work when the spec actually changes. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
The `HashAllocator` uses a go map to store metadata for allocated IPs. The go runtime automatically grows the map when it becomes full. However, it does not automatically shrink the map when it becomes empty. So when a large number of IPs were allocated at some point and then deallocated, the map would hold onto more memory than necessary. This commit adds logic to the `HashAllocator` to re-create the map when its underutilized. Go does not allow us to query its "capacity" so we track its peak size instead. When the current number of allocated IPs is less than 1/3 of the peak size, we re-create the map. The new map gets a size indication of 1/2 the peak size. This leaves some room for growth before the map needs to be grown again. During testing we allocated 50k IPs, then deallocated all. This change reduced the resident memory usage by 50MB compared to not re-creating the map. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
|
/test |
doniacld
approved these changes
May 5, 2025
|
@cilium/sig-lb could I get a review on this? |
joamaki
approved these changes
May 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses two issues that were surfaced by scale testing as part of #39276.
Please see the individual commit messages for details.