lib/promscrape: added label_limit global and per job scrape param #7795
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
e130c6d to
1e01be3
Compare
This was
linked to
issues
Feb 17, 2025
f41gh7
reviewed
Mar 11, 2025
f41gh7
reviewed
Mar 11, 2025
f41gh7
reviewed
Mar 11, 2025
f41gh7
reviewed
Mar 11, 2025
9a6cd6d to
642d074
Compare
hagen1778
requested changes
May 29, 2025
|
@AndrewChubatiuk kindly pinging you on this |
5250811 to
a8d1d74
Compare
602cb6c to
f213ad6
Compare
|
@hagen1778 please review it |
hagen1778
requested changes
Jun 19, 2025
36365b0 to
6b2a9f6
Compare
f41gh7
reviewed
Jun 27, 2025
f41gh7
reviewed
Jun 27, 2025
f41gh7
reviewed
Jun 27, 2025
f41gh7
reviewed
Jun 27, 2025
lib/promscrape/scrapework.go
Outdated
| dst = appendRow(dst, "scrape_timeout_seconds", sw.Config.ScrapeTimeout.Seconds(), timestamp) | ||
| dst = appendRow(dst, "up", float64(am.up), timestamp) | ||
|
|
||
| wc.addRows(sw.Config, dst, timestamp, false) | ||
| _ = wc.tryAddRows(sw.Config, dst, timestamp, false) |
There was a problem hiding this comment.
If it's impossible case, please add assert: logger.Fatalf("BUG: tryAddRows cannot return error for autoMetrics.
There was a problem hiding this comment.
Looks like, it's possible to breach this limit and get panic.
There is a problem, we can either: apply labels limit to auto-metrics as well and log it as error. Or ignore labels limit for it.
I think we should go with 1 option. Next hope storage can apply own limits and drop auto metrics as well.
There was a problem hiding this comment.
apply labels limit to auto-metrics as well and log it as error
did this. see updated
f41gh7
reviewed
Jun 27, 2025
cb24497 to
3eaebd0
Compare
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* remove double update of `scrapesSkippedByLabelLimit` on L:1032 * make processing of label_limit and sample_limit alike * process error on stale series send, as it would give at least some hints on what's going on. Before, the stale series will be just silently dropped. Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
…for consistency with `scrape_samples_limit` Signed-off-by: hagen1778 <roman@victoriametrics.com>
…ted metrics Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Max Kotliar <kotlyar.maksim@gmail.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
limit of labels amount can be exceeded when adding auto metrics, so it is a bad idea to panic here Signed-off-by: hagen1778 <roman@victoriametrics.com>
d999488 to
6771c84
Compare
AndrewChubatiuk
commented
Jul 4, 2025
Co-authored-by: Andrii Chubatiuk <achubatiuk@victoriametrics.com>
|
Thanks for contribution! |
f41gh7
pushed a commit
that referenced
this pull request
Jul 4, 2025
) This commit introduces new scrape config option - `label_limit`. Which defines labels count limit for time series. Whole scrape will be discarded In case of limit breach. This option could also be defined with label `__label_limit__` during scrape relabeling process and globally for all scrape targets. Fixes #7660 and #3233
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe Your Changes
related issues:
label_limitin scrape config on job level #7660with this change suggestion for #3233 is to
label_limitto a default value for scrape configurationlabel_limitwith a value, that is up to maxLabelsPerTimeseries if it's definedthis will allow avoid propagation of custom labelLimits into tryPush agent's function
Checklist
The following checks are mandatory: