Add scale test for lb-ipam #39276
Merged
Add scale test for lb-ipam #39276
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
e0249e1 to
fcb9e5b
Compare
|
/test |
fcb9e5b to
25135e2
Compare
|
/scale-100 |
25135e2 to
925ed0f
Compare
In preparation of scale testing, we added metrics to track the time it takes to process events in the lbipam controller. This will help give us insight into how LB-IPAM performance scales with the number of services in the cluster. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
This commit introduces a basic scale test module for lb-ipam. The module will create a LB-IPAM pool, and create 50k services in 3 minutes. It measures the 50, 90 and 99 percentiles of the time it takes to process a service, as well as the max CPU and memory usage of the operator during this time. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
925ed0f to
ff937d1
Compare
|
/scale-5 |
ff937d1 to
4c3b1a5
Compare
d3df317 to
b786d6d
Compare
|
Alright. Got the new workflow to run, seems successful: https://github.com/cilium/cilium/actions/runs/15184456179/job/42701510501 Will disable test run on push now, then mark as ready for review. |
b786d6d to
85ca550
Compare
joestringer
reviewed
May 22, 2025
There was a problem hiding this comment.
It would be nice if we could share more of the core content here by defining some reusable workflow for both scale test workflows, and parameterize the node count and testconfig parameters.
The full diff is very small compared to 100 node test. I glanced through the workflow and it seems sane overall. A couple of minor comments below.
christarazi
approved these changes
May 22, 2025
julianwiedmann
approved these changes
May 26, 2025
thorn3r
approved these changes
Jun 3, 2025
The LB-IPAM scale test is relatively small and only puts load on the operator. So adding a small cluster test with only 5 nodes which will be a lot cheaper then adding to any of the other scale test workflows. Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
85ca550 to
3e3d1d7
Compare
|
/test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a scale test of LB-IPAM to prove that the feature scales well event when a lot of services are created. The test is fairly simple, we create a pool and then over the course of 3 minutes (by default) add 50k services which get an IP from the pool. We record the event processing time, CPU usage and memory usage.