-
Notifications
You must be signed in to change notification settings - Fork 527
Set minAllowed CPU for prometheus-shoot to avoid frequent evictions #12054
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Set minAllowed CPU for prometheus-shoot to avoid frequent evictions #12054
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
LGTM label has been added. Git tree hash: 4ccfc4c144fdf84c9acb3dd0002a6a55233cb51f
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/approve
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rfranzke The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/cherry-pick release-v1.118 |
@voelzmo: new pull request created: #12069 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
The robot seems to process only 1 |
/cherry-pick release-v1.117 |
/cherry-pick release-v1.116 |
@istvanballok: new pull request created: #12079 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
@istvanballok: new pull request created: #12080 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
How to categorize this PR?
/area auto-scaling
/area monitoring
/kind enhancement
What this PR does / why we need it:
This PR sets a
minAllowed
CPU value for prometheus-shoot to150mCores
. We have seen way too frequent evictions on theprometheus-shoot
instances for very small absolute CPU values (e.g ~20 changes in 24 hours for moving between70mCores
and130mCores
in intermediate steps of~20mCores
and back again).We want to address this upstream by adjusting how the vpa computes the upper and lower bounds around the
target
recommendation. In theory, this is the mechanism that should take care of not evicting for very small changes. In practice, however, thelowerBound
andupperBound
are way too close to thetarget
, sometimes even identical.So this is a temporary fix to spend more resources in order to avoid too frequent evictions.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
The PR also adds
100M
for memory to theminAllowed
structure, in order to keep the existing behavior. Currently, no values are defined forVPAMinAllowed
, therefore it is defaulted tomemory: 100M
.Release note: