Skip to content

Conversation

SuperQ
Copy link
Member

@SuperQ SuperQ commented Jun 1, 2024

Add the ability to adjust the GOGC variable from the Prometheus configuration file.

  • Create a new top-level runtime section in the config.
  • Adjust from the Go default of 100 to 50 to reduce wasted memory.
  • Use the GOGC env value if no configuraiton is used.

@SuperQ SuperQ requested a review from bboreham June 1, 2024 20:45
@SuperQ SuperQ force-pushed the superq/gogc branch 3 times, most recently from 080fdf1 to 7e2a3cb Compare June 1, 2024 21:54
@bboreham bboreham changed the title Add configuration option for GOGC Add configuration option for GOGC, default to 50 Jun 3, 2024
Copy link
Member

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok with this; seems a bit annoying to have to parse the env var ourselves but I guess you looked into that.

The change to the default should be called out in the release notes, since it will provoke a rise in CPU usage and some people might have lots of free memory but little CPU. And we should write some explanatory text what is happening, since my experience is most people don't know GOGC or the basic mechanism.

@roidelapluie
Copy link
Member

Use the GOGC env value if no configuraiton is used.

This is the opposite?

@SuperQ
Copy link
Member Author

SuperQ commented Jun 3, 2024

Yes, the Env var parsing is having to reproduce what Go does internally since this isn't exposed. The source is here.

@SuperQ
Copy link
Member Author

SuperQ commented Jun 3, 2024

@roidelapluie The logic flow is a bit hard to follow, let me see if I can refactor it a bit to make it more clear.

Copy link
Member

@machine424 machine424 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we don’t allow automatically setting it like for gomaxprocs or gomemlimit, why do you think that using the GOGC environment variable isn’t adequate/sufficient? I would maintain the default value at 100 (if the environment variable isn’t set), as we can’t guarantee that 50 won’t degrade or cause issues in any setup. We should allow users, who need it, the discretion to select the value that suits them best.

@SuperQ
Copy link
Member Author

SuperQ commented Jun 3, 2024

@machine424 The reason we want to change the default is that several of us have done reasonably extensive production testing that shows that the additional CPU use for Prometheus is pretty minimal.

Especially as we've made major improvements in the overall rate of allocs over the last few years. If I look at metrics history over the last 2 years, the peak alloc bytes/sec/series is 30-40% of what it used to be.

By reducing the Go default from 100 to 50, we see a significant drop in RSS memory use.

@SuperQ SuperQ force-pushed the superq/gogc branch 5 times, most recently from 30662f3 to d0c139d Compare June 3, 2024 16:00
@machine424
Copy link
Member

e424 The reason we want to change the default is that several of us have done reasonably extensive production testing that shows that the additional CPU use for Prometheus is pretty minimal.

Especially as we've made major improvements in the overall rate of allocs over the last few years. If I look at metrics history over the last 2 years, the peak alloc bytes/sec/series is 30-40% of what it used to be.

By reducing the Go default from 100 to 50, we see a significant drop in RSS memory use.

Thanks for the context and the details, this looks promising.
Ok, let's give it a try, and if we start yelling about it, we can always adjust or rollback ;)

@SuperQ SuperQ force-pushed the superq/gogc branch 3 times, most recently from 51e9080 to a4be027 Compare June 5, 2024 08:29
@SuperQ
Copy link
Member Author

SuperQ commented Jun 5, 2024

I've refactored this to create a new top-level runtime section.

@SuperQ SuperQ force-pushed the superq/gogc branch 2 times, most recently from 70b350c to 73e6b1b Compare June 5, 2024 08:58
Copy link
Member

@machine424 machine424 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks.

@SuperQ
Copy link
Member Author

SuperQ commented Jun 5, 2024

Moving the configuration to a separate top level function seems to break the default env var flow. The UnmarshalYAML() seems to not be called if there is no runtime top level key in the config.

@SuperQ
Copy link
Member Author

SuperQ commented Jun 5, 2024

Ok, I figured out how to get the empty runtime section to work with the ENV var set.

Add the ability to adjust the `GOGC` variable from the Prometheus
configuration file.
* Create a new top-level `runtime` section in the config.
* Adjust from the Go default of 100 to 50 to reduce wasted memory.
* Use the `GOGC` env value if no configuraiton is used.

Signed-off-by: SuperQ <superq@gmail.com>
@machine424
Copy link
Member

machine424 commented Jun 5, 2024

Good catch!, I’d ask for a regression test for that case, but I think you’ve had enough :).
I’ll add some when I have some time ;)
lgtm

@SuperQ SuperQ merged commit 4e66403 into main Jun 5, 2024
@SuperQ SuperQ deleted the superq/gogc branch June 5, 2024 15:43
machine424 added a commit to machine424/prometheus that referenced this pull request Oct 29, 2024
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request Oct 29, 2024
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request Mar 27, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request Mar 27, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request Apr 23, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request Apr 23, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request Apr 23, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request May 1, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request May 4, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request May 6, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request May 6, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit to machine424/prometheus that referenced this pull request May 6, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
machine424 added a commit that referenced this pull request May 6, 2025
As suggested in #14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
Shuimo03 pushed a commit to Shuimo03/prometheus that referenced this pull request May 8, 2025
As suggested in prometheus#14176 (comment)

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
Signed-off-by: WuLiang <wu1998102@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants