-
Notifications
You must be signed in to change notification settings - Fork 9.8k
Add configuration option for GOGC, default to 50 #14176
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
080fdf1
to
7e2a3cb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm ok with this; seems a bit annoying to have to parse the env var ourselves but I guess you looked into that.
The change to the default should be called out in the release notes, since it will provoke a rise in CPU usage and some people might have lots of free memory but little CPU. And we should write some explanatory text what is happening, since my experience is most people don't know GOGC or the basic mechanism.
Use the GOGC env value if no configuraiton is used. This is the opposite? |
Yes, the Env var parsing is having to reproduce what Go does internally since this isn't exposed. The source is here. |
@roidelapluie The logic flow is a bit hard to follow, let me see if I can refactor it a bit to make it more clear. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we don’t allow automatically setting it like for gomaxprocs or gomemlimit, why do you think that using the GOGC environment variable isn’t adequate/sufficient? I would maintain the default value at 100 (if the environment variable isn’t set), as we can’t guarantee that 50 won’t degrade or cause issues in any setup. We should allow users, who need it, the discretion to select the value that suits them best.
@machine424 The reason we want to change the default is that several of us have done reasonably extensive production testing that shows that the additional CPU use for Prometheus is pretty minimal. Especially as we've made major improvements in the overall rate of allocs over the last few years. If I look at metrics history over the last 2 years, the peak alloc bytes/sec/series is 30-40% of what it used to be. By reducing the Go default from 100 to 50, we see a significant drop in RSS memory use. |
30662f3
to
d0c139d
Compare
Thanks for the context and the details, this looks promising. |
51e9080
to
a4be027
Compare
I've refactored this to create a new top-level |
70b350c
to
73e6b1b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm, thanks.
Moving the configuration to a separate top level function seems to break the default env var flow. The |
Ok, I figured out how to get the empty runtime section to work with the ENV var set. |
Add the ability to adjust the `GOGC` variable from the Prometheus configuration file. * Create a new top-level `runtime` section in the config. * Adjust from the Go default of 100 to 50 to reduce wasted memory. * Use the `GOGC` env value if no configuraiton is used. Signed-off-by: SuperQ <superq@gmail.com>
Good catch!, I’d ask for a regression test for that case, but I think you’ve had enough :). |
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in #14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com>
As suggested in prometheus#14176 (comment) Signed-off-by: machine424 <ayoubmrini424@gmail.com> Signed-off-by: WuLiang <wu1998102@gmail.com>
Add the ability to adjust the
GOGC
variable from the Prometheus configuration file.runtime
section in the config.GOGC
env value if no configuraiton is used.