tsdb: add enable overlapping compaction #13393
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SuperQ
reviewed
Jan 12, 2024
This functionality is needed in downstream projects because they have a separate component that does compaction. Upstreaming https://github.com/grafana/mimir-prometheus/blob/7c8e9a2a76fc729e9078889782928b2fdfe240e9/tsdb/compact.go#L323-L325. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
SuperQ
approved these changes
Jan 12, 2024
GiedriusS
added a commit
to thanos-io/thanos
that referenced
this pull request
Jan 12, 2024
This reverts commit 7b8eb86. Proper way to handle this is to disable vertical compaction. I am trying to add this functionality here: prometheus/prometheus#13393 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
pracucci
approved these changes
Jan 12, 2024
yeya24
pushed a commit
to thanos-io/thanos
that referenced
this pull request
Jan 12, 2024
This reverts commit 7b8eb86. Proper way to handle this is to disable vertical compaction. I am trying to add this functionality here: prometheus/prometheus#13393 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
|
Previous change in #13282 |
|
Yes, sorry for the churn. I didn't realize what this flag was for until I went down the rabbit hole. |
|
Is it worth addding a note for the next person? The previous note (which was not sufficient) read: |
jesusvazquez
pushed a commit
that referenced
this pull request
Jan 15, 2024
This added the #13393 "EnableOverlappingCompaction" parameter to the compactor code but not to the tsdb.Options. I forgot about that. Add it to `tsdb.Options` too and set it to `true` in Prometheus. Copy/paste the description from #13393 (comment) Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
GiedriusS
added a commit
to vinted/thanos
that referenced
this pull request
Jan 18, 2024
…)" (thanos-io#7053) This reverts commit 7b8eb86. Proper way to handle this is to disable vertical compaction. I am trying to add this functionality here: prometheus/prometheus#13393 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Merged
3 tasks
hanyuting8
pushed a commit
to hanyuting8/thanos
that referenced
this pull request
Jan 19, 2024
…)" (thanos-io#7053) This reverts commit 7b8eb86. Proper way to handle this is to disable vertical compaction. I am trying to add this functionality here: prometheus/prometheus#13393 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: hanyuting8 <hytxidian@163.com>
hanyuting8
pushed a commit
to hanyuting8/thanos
that referenced
this pull request
Jan 19, 2024
…)" (thanos-io#7053) This reverts commit 7b8eb86. Proper way to handle this is to disable vertical compaction. I am trying to add this functionality here: prometheus/prometheus#13393 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: hanyuting8 <hytxidian@163.com>
cstyan
pushed a commit
that referenced
this pull request
Feb 2, 2024
* consoles: exclude iowait and steal from CPU Utilisation 'iowait' and 'steal' indicate specific idle/wait states, which shouldn't be counted into CPU Utilisation. Also see prometheus-operator/kube-prometheus#796 and kubernetes-monitoring/kubernetes-mixin#667. Per the iostat man page: %idle Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request. %iowait Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request. %steal Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor. Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com> * tsdb: shrink txRing with smaller integers 4 billion active transactions ought to be enough for anyone. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * tsdb: create isolation transaction slice on demand When Prometheus restarts it creates every series read in from the WAL, but many of those series will be finished, and never receive any more samples. By defering allocation of the txRing slice to when it is first needed, we save 32 bytes per stale series. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * add cluster variable to Overview dashboard Signed-off-by: Erik Sommer <ersotech@posteo.de> * promql: simplify Native Histogram arithmetics Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com> * Cut 2.49.0-rc.0 (#13270) * Cut 2.49.0-rc.0 Signed-off-by: bwplotka <bwplotka@gmail.com> * Removed the duplicate. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> * Add unit protobuf parser Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * Go on adding protobuf parsing for unit Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * ui: create a reproduction for #13292 Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Get conditional right Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * Get VM Scale Set NIC (#13283) Calling `*armnetwork.InterfacesClient.Get()` doesn't work for Scale Set VM NIC, because these use a different Resource ID format. Use `*armnetwork.InterfacesClient.GetVirtualMachineScaleSetNetworkInterface()` instead. This needs both the scale set name and the instance ID, so add an `InstanceID` field to the `virtualMachine` struct. `InstanceID` is empty for a VM that isn't a ScaleSetVM. Signed-off-by: Daniel Nicholls <daniel.nicholls@resdiary.com> * Cut v2.49.0-rc.1 Signed-off-by: bwplotka <bwplotka@gmail.com> * Delete debugging lines, amend error message for unit Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * Correct order in error message Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * Consider storage.ErrTooOldSample as non-retryable Signed-off-by: Daniel Kerbel <nmdanny@gmail.com> * scrape_test.go: Increase scrape interval in TestScrapeLoopCache to reduce potential flakiness Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Avoid creating string for suffix, consider counters without _total suffix Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * build(deps): bump github.com/prometheus/client_golang Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.17.0 to 1.18.0. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](prometheus/client_golang@v1.17.0...v1.18.0) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump actions/setup-node from 3.8.1 to 4.0.1 Bumps [actions/setup-node](https://github.com/actions/setup-node) from 3.8.1 to 4.0.1. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@5e21ff4...b39b52d) --- updated-dependencies: - dependency-name: actions/setup-node dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * scripts: sort file list in embed directive Otherwise the resulting string depends on find, which afaict depends on the underlying filesystem. A stable file list make it easier to detect UI changes in downstreams that need to track UI assets. Signed-off-by: Jan Fajerski <jfajersk@redhat.com> * Fix DataTableProps['data'] for resultType string Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> * Fix handling of scalar and string in isHeatmapData Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> * build(deps): bump github.com/influxdata/influxdb Bumps [github.com/influxdata/influxdb](https://github.com/influxdata/influxdb) from 1.11.2 to 1.11.4. - [Release notes](https://github.com/influxdata/influxdb/releases) - [Commits](influxdata/influxdb@v1.11.2...v1.11.4) --- updated-dependencies: - dependency-name: github.com/influxdata/influxdb dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.48.0 to 0.48.1. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](v0.48.0...v0.48.1) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump client_golang to v1.18.0 (#13373) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Drop old inmemory samples (#13002) * Drop old inmemory samples Co-authored-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Avoid copying timeseries when the feature is disabled Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Run gofmt Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Clarify docs Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Add more logging info Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Remove loggers Signed-off-by: Marc Tuduri <marctc@protonmail.com> * optimize function and add tests Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Simplify filter Signed-off-by: Marc Tuduri <marctc@protonmail.com> * rename var Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Update help info from metrics Signed-off-by: Marc Tuduri <marctc@protonmail.com> * use metrics to keep track of drop elements during buildWriteRequest Signed-off-by: Marc Tuduri <marctc@protonmail.com> * rename var in tests Signed-off-by: Marc Tuduri <marctc@protonmail.com> * pass time.Now as parameter Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Change buildwriterequest during retries Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Revert "Remove loggers" This reverts commit 54f91df. Signed-off-by: Marc Tuduri <marctc@protonmail.com> * use log level debug for loggers Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Fix linter Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove noisy debug-level logs; add 'reason' label to drop metrics Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove accidentally committed files Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Propagate logger to buildWriteRequest to log dropped data Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Fix docs comment Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Make drop reason more specific Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove unnecessary pass of logger Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Use snake_case for reason label Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Fix dropped samples metric Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> --------- Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> Signed-off-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> * fix(discovery): allow requireUpdate util to timeout in discovery/file/file_test.go. The loop ran indefinitely if the condition isn't met. Before, each iteration created a new timer channel which was always outpaced by the other timer channel with smaller duration. minor detail: There was a memory leak: resources of the ~10 previous timers were constantly kept. With the fix, we may keep the resources of one timer around for defaultWait but this isn't worth the changes to make it right. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Merge pull request #13371 from kevinmingtarja/fix-isHeatmapData ui: fix handling of scalar and string in isHeatmapData * tsdb/{index,compact}: allow using custom postings encoding format (#13242) * tsdb/{index,compact}: allow using custom postings encoding format We would like to experiment with a different postings encoding format in Thanos so in this change I am proposing adding another argument to `NewWriter` which would allow users to change the format if needed. Also, wire the leveled compactor so that it would be possible to change the format there too. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb/compact: use a struct for leveled compactor options As discussed on Slack, let's use a struct for the options in leveled compactor. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb: make changes after Bryan's review - Make changes less intrusive - Turn the postings encoder type into a function - Add NewWriterWithEncoder() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Cut 2.49.0-rc.2 Signed-off-by: bwplotka <bwplotka@gmail.com> * build(deps): bump actions/setup-go from 3.5.0 to 5.0.0 in /scripts (#13362) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3.5.0 to 5.0.0. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@6edd440...0c52d54) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github/codeql-action from 2.22.8 to 3.22.12 (#13358) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.22.8 to 3.22.12. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@407ffaf...012739e) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * put @Nexucis has a release shepherd (#13383) Signed-off-by: Augustin Husson <augustin.husson@amadeus.com> * Add analyze histograms command to promtool (#12331) Add `query analyze` command to promtool This command analyzes the buckets of classic and native histograms, based on data queried from the Prometheus query API, i.e. it doesn't require direct access to the TSDB files. Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> * included instance in all necessary descriptions Signed-off-by: Erik Sommer <ersotech@posteo.de> * tsdb/compact: fix passing merge func Fixing a very small logical problem I've introduced :(. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb: add enable overlapping compaction This functionality is needed in downstream projects because they have a separate component that does compaction. Upstreaming https://github.com/grafana/mimir-prometheus/blob/7c8e9a2a76fc729e9078889782928b2fdfe240e9/tsdb/compact.go#L323-L325. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Cut 2.49.0 Signed-off-by: bwplotka <bwplotka@gmail.com> * promtool: allow setting multiple matchers to "promtool tsdb dump" command. (#13296) Conditions are ANDed inside the same matcher but matchers are ORed Including unit tests for "promtool tsdb dump". Refactor some matchers scraping utils. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Fixed changelog Signed-off-by: bwplotka <bwplotka@gmail.com> * tsdb/main: wire "EnableOverlappingCompaction" to tsdb.Options (#13398) This added the #13393 "EnableOverlappingCompaction" parameter to the compactor code but not to the tsdb.Options. I forgot about that. Add it to `tsdb.Options` too and set it to `true` in Prometheus. Copy/paste the description from #13393 (comment) Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Issue #13268: fix quality value in accept header Signed-off-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com> * Cut 2.49.1 with scrape q= bugfix. Signed-off-by: bwplotka <bwplotka@gmail.com> * Cut 2.49.1 web package. Signed-off-by: bwplotka <bwplotka@gmail.com> * Restore more efficient version of NewPossibleNonCounterInfo annotation (#13022) Restore more efficient version of NewPossibleNonCounterInfo annotation Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> * Fix regressions introduced by #13242 Signed-off-by: Marco Pracucci <marco@pracucci.com> * fix slice copy in 1.20 (#13389) The slices package is added to the standard library in Go 1.21; we need to import from the exp area to maintain compatibility with Go 1.20. Signed-off-by: tyltr <tylitianrui@126.com> * Docs: Query Basics: link to rate (#10538) Co-authored-by: Julien Pivotto <roidelapluie@o11y.eu> * chore(kubernetes): check preconditions earlier and avoid unnecessary checks or iterations Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Examples: link to `rate` for new users (#10535) * Examples: link to `rate` for new users Signed-off-by: Ted Robertson 10043369+tredondo@users.noreply.github.com Co-authored-by: Bryan Boreham <bjboreham@gmail.com> * promql: use natural sort in sort_by_label and sort_by_label_desc (#13411) These functions are intended for humans, as robots can already sort the results however they please. Humans like things sorted "naturally": * https://blog.codinghorror.com/sorting-for-humans-natural-sort-order/ A similar thing has been done to Grafana, which is also used by humans: * grafana/grafana#78024 * grafana/grafana#78494 Signed-off-by: Ivan Babrou <github@ivan.computer> * TestLabelValuesWithMatchers: Add test case Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * remove obsolete build tag Signed-off-by: tyltr <tylitianrui@126.com> * Upgrade some golang dependencies for resty 2.11 Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Native Histograms: support `native_histogram_min_bucket_factor` in scrape_config (#13222) Native Histograms: support native_histogram_min_bucket_factor in scrape_config --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> * Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram (#13392) Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> * Minor fixes to otlp vendor update script Signed-off-by: Goutham <gouthamve@gmail.com> * build(deps): bump github.com/hetznercloud/hcloud-go/v2 Bumps [github.com/hetznercloud/hcloud-go/v2](https://github.com/hetznercloud/hcloud-go) from 2.4.0 to 2.6.0. - [Release notes](https://github.com/hetznercloud/hcloud-go/releases) - [Changelog](https://github.com/hetznercloud/hcloud-go/blob/main/CHANGELOG.md) - [Commits](hetznercloud/hcloud-go@v2.4.0...v2.6.0) --- updated-dependencies: - dependency-name: github.com/hetznercloud/hcloud-go/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Enhanced visibility for `promtool test rules` with JSON colored formatting (#13342) * Added diff flag for unit test to improvise readability & debugging Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Removed blank spaces Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Fixed linting error Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Added cli flags to documentation Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Revert unrrelated linting fixes Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Fixed review suggestions Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Cleanup Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Updated flag description Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Updated flag description Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> --------- Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * storage: skip merging when no remote storage configured Prometheus is hard-coded to use a fanout storage between TSDB and a remote storage which by default is empty. This change detects the empty storage and skips merging between result sets, which would make `Select()` sort results. Bottom line: we skip a sort unless there really is some remote storage configured. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Remove csmarchbanks from remote write owners (#13432) I have not had the time to keep up with remote write and have no plans to work on it in the near future so I am withdrawing my maintainership of that part of the codebase. I continue to focus on client_python. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com> * add more context cancellation check at evaluation time Signed-off-by: Ben Ye <benye@amazon.com> * Optimize label values with matchers by taking shortcuts (#13426) Don't calculate postings beforehand: we may not need them. If all matchers are for the requested label, we can just filter its values. Also, if there are no values at all, no need to run any kind of logic. Also add more labelValuesWithMatchers benchmarks Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Add automatic memory limit handling Enable automatic detection of memory limits and configure GOMEMLIMIT to match. * Also includes a flag to allow controlling the reserved ratio. Signed-off-by: SuperQ <superq@gmail.com> * Update OSSF badge link (#13433) Provide a more user friendly interface Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> * SD Managers taking over responsibility for registration of debug metrics (#13375) SD Managers take over responsibility for SD metrics registration --------- Signed-off-by: Paulin Todev <paulin.todev@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Björn Rabenstein <github@rabenste.in> * Optimize histogram iterators (#13340) Optimize histogram iterators Histogram iterators allocate new objects in the AtHistogram and AtFloatHistogram methods, which makes calculating rates over long ranges expensive. In #13215 we allowed an existing object to be reused when converting an integer histogram to a float histogram. This commit follows the same idea and allows injecting an existing object in the AtHistogram and AtFloatHistogram methods. When the injected value is nil, iterators allocate new histograms, otherwise they populate and return the injected object. The commit also adds a CopyTo method to Histogram and FloatHistogram which is used in the BufferedIterator to overwrite items in the ring instead of making new copies. Note that a specialized HPoint pool is needed for all of this to work (`matrixSelectorHPool`). --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> * doc: Mark `mad_over_time` as experimental (#13440) We forgot to do that in #13059 Signed-off-by: beorn7 <beorn@grafana.com> * Change metric label for Puppetdb from 'http' to 'puppetdb' Signed-off-by: Paulin Todev <paulin.todev@gmail.com> * mirror metrics.proto change & generate code Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> * TestHeadLabelValuesWithMatchers: Add test case (#13414) Add test case to TestHeadLabelValuesWithMatchers, while fixing a couple of typos in other test cases. Also enclosing some implicit sub-tests in a `t.Run` call to make them explicitly sub-tests. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * update all go dependencies (#13438) Signed-off-by: Augustin Husson <husson.augustin@gmail.com> * build(deps): bump the k8s-io group with 2 updates (#13454) Bumps the k8s-io group with 2 updates: [k8s.io/api](https://github.com/kubernetes/api) and [k8s.io/client-go](https://github.com/kubernetes/client-go). Updates `k8s.io/api` from 0.28.4 to 0.29.1 - [Commits](kubernetes/api@v0.28.4...v0.29.1) Updates `k8s.io/client-go` from 0.28.4 to 0.29.1 - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.28.4...v0.29.1) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s-io - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s-io ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump the go-opentelemetry-io group with 1 update (#13453) Bumps the go-opentelemetry-io group with 1 update: [go.opentelemetry.io/collector/semconv](https://github.com/open-telemetry/opentelemetry-collector). Updates `go.opentelemetry.io/collector/semconv` from 0.92.0 to 0.93.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-collector/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG-API.md) - [Commits](open-telemetry/opentelemetry-collector@v0.92.0...v0.93.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/collector/semconv dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump actions/upload-artifact from 3.1.3 to 4.0.0 (#13355) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.3 to 4.0.0. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@a8a3f3a...c7d193f) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump bufbuild/buf-push-action (#13357) Bumps [bufbuild/buf-push-action](https://github.com/bufbuild/buf-push-action) from 342fc4cdcf29115a01cf12a2c6dd6aac68dc51e1 to a654ff18effe4641ebea4a4ce242c49800728459. - [Release notes](https://github.com/bufbuild/buf-push-action/releases) - [Commits](bufbuild/buf-push-action@342fc4c...a654ff1) --- updated-dependencies: - dependency-name: bufbuild/buf-push-action dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Labels: Add DropMetricName function, used in PromQL (#13446) This function is called very frequently when executing PromQL functions, and we can do it much more efficiently inside Labels. In the common case that `__name__` comes first in the labels, we simply re-point to start at the next label, which is nearly free. `DropMetricName` is now so cheap I removed the cache - benchmarks show everything still goes faster. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * tsdb: simplify internal series delete function (#13261) Lifting an optimisation from Agent code, `seriesHashmap.del` can use the unique series reference, doesn't need to check Labels. Also streamline the logic for deleting from `unique` and `conflicts` maps, and add some comments to help the next person. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * otlptranslator/update-copy.sh: Fix sed command lines Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Rollback k8s.io requirements (#13462) Rollback k8s.io Go modules to v0.28.6 to avoid forcing upgrade of Go to 1.21. This allows us to keep compatibility with the currently supported upstream Go releases. Signed-off-by: SuperQ <superq@gmail.com> * Make update-copy.sh work for both OSX and GNU sed Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Name @beorn7 and @krajorama as maintainers for native histograms I have been the de-facto maintainer for native histograms from the beginning. So let's put this into MAINTAINERS.md. In addition, I hereby proposose George Krajcsovits AKA Krajo as a co-maintainer. He has contributed a lot of native histogram code, but more importantly, he has contributed substantially to reviewing other contributors' native histogram code, up to a point where I was merely rubberstamping the PRs he had already reviewed. I'm confident that he is ready to to be granted commit rights as outlined in the "Maintainers" section of the governance: https://prometheus.io/governance/#maintainers According to the same section of the governance, I will announce the proposed change on the developers mailing list and will give some time for lazy consensus before merging this PR. Signed-off-by: beorn7 <beorn@grafana.com> * ui/fix: correct url handling for stacked graphs (#13460) Signed-off-by: Yury Moladau <yurymolodov@gmail.com> * tsdb: use cheaper Mutex on series Mutex is 8 bytes; RWMutex is 24 bytes and much more complicated. Since `RLock` is only used in two places, `UpdateMetadata` and `Delete`, neither of which are hotspots, we should use the cheaper one. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Fix last_over_time for native histograms The last_over_time retains a histogram sample without making a copy. This sample is now coming from the buffered iterator used for windowing functions, and can be reused for reading subsequent samples as the iterator progresses. I would propose copying the sample in the last_over_time function, similar to how it is done for rate, sum_over_time and others. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Implementation NOTE: Rebased from main after refactor in #13014 Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Add feature flag Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactor concurrency control Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Optimising dependencies/dependents funcs to not produce new slices each request Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Rename flag Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring for performance, and to allow controller to be overridden Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Block until all rules, both sync & async, have completed evaluating Updated & added tests Review feedback nits Return empty map if not indeterminate Use highWatermark to track inflight requests counter Appease the linter Clarify feature flag Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fix typo in CLI flag description Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed auto-generated doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Improve doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Simplify the design to update concurrency controller once the rule evaluation has done Signed-off-by: Marco Pracucci <marco@pracucci.com> * Add more test cases to TestDependenciesEdgeCases Signed-off-by: Marco Pracucci <marco@pracucci.com> * Added more test cases to TestDependenciesEdgeCases Signed-off-by: Marco Pracucci <marco@pracucci.com> * Improved RuleConcurrencyController interface doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Introduced sequentialRuleEvalController Signed-off-by: Marco Pracucci <marco@pracucci.com> * Remove superfluous nil check in Group.metrics Signed-off-by: Marco Pracucci <marco@pracucci.com> * api: Serialize discovered and target labels into JSON directly (#13469) Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> * api: Serialize discovered labels into JSON directly in dropped targets (#13484) Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> * Add ShardedPostings() support to TSDB (#10421) This PR is a reference implementation of the proposal described in #10420. In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing). Follow up work As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes. Signed-off-by: Marco Pracucci <marco@pracucci.com> * storage/remote: document why two benchmarks are skipped One was silently doing nothing; one was doing something but the work didn't go up linearly with iteration count. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Pod status changes not discovered by Kube Endpoints SD (#13337) * fix(discovery/kubernetes/endpoints): react to changes on Pods because some modifications can occur on them without triggering an update on the related Endpoints (The Pod phase changing from Pending to Running e.g.). --------- Signed-off-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: Guillermo Sanchez Gavier <gsanchez@newrelic.com> * Small improvements, add const, remove copypasta (#8106) Signed-off-by: Mikhail Fesenko <proggga@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> * Proposal to improve FPointSlice and HPointSlice allocation. (#13448) * Reusing points slice from previous series when the slice is under utilized * Adding comments on the bench test Signed-off-by: Alan Protasio <alanprot@gmail.com> * lint Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * go mod tidy Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> --------- Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Erik Sommer <ersotech@posteo.de> Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> Signed-off-by: machine424 <ayoubmrini424@gmail.com> Signed-off-by: Daniel Nicholls <daniel.nicholls@resdiary.com> Signed-off-by: Daniel Kerbel <nmdanny@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Jan Fajerski <jfajersk@redhat.com> Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> Signed-off-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: Augustin Husson <augustin.husson@amadeus.com> Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com> Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: tyltr <tylitianrui@126.com> Signed-off-by: Ted Robertson 10043369+tredondo@users.noreply.github.com Signed-off-by: Ivan Babrou <github@ivan.computer> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: Israel Blancas <iblancasa@gmail.com> Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Goutham <gouthamve@gmail.com> Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: SuperQ <superq@gmail.com> Signed-off-by: Ben Kochie <superq@gmail.com> Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Paulin Todev <paulin.todev@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: beorn7 <beorn@grafana.com> Signed-off-by: Augustin Husson <husson.augustin@gmail.com> Signed-off-by: Yury Moladau <yurymolodov@gmail.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> Signed-off-by: Mikhail Fesenko <proggga@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> Co-authored-by: Julian Wiedmann <jwi@linux.ibm.com> Co-authored-by: Bryan Boreham <bjboreham@gmail.com> Co-authored-by: Erik Sommer <ersotech@posteo.de> Co-authored-by: Linas Medziunas <linas.medziunas@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Arianna Vespri <arianna.vespri@yahoo.it> Co-authored-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: daniel-resdiary <109083091+daniel-resdiary@users.noreply.github.com> Co-authored-by: Daniel Kerbel <nmdanny@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jan Fajerski <jfajersk@redhat.com> Co-authored-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: Marc Tudurí <marctc@protonmail.com> Co-authored-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Co-authored-by: Augustin Husson <husson.augustin@gmail.com> Co-authored-by: Björn Rabenstein <beorn@grafana.com> Co-authored-by: zenador <zenador@users.noreply.github.com> Co-authored-by: gotjosh <josue.abreu@gmail.com> Co-authored-by: Ben Kochie <superq@gmail.com> Co-authored-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com> Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: tyltr <tylitianrui@126.com> Co-authored-by: Ted Robertson <10043369+tredondo@users.noreply.github.com> Co-authored-by: Julien Pivotto <roidelapluie@o11y.eu> Co-authored-by: Matthias Loibl <mail@matthiasloibl.com> Co-authored-by: Ivan Babrou <github@ivan.computer> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Israel Blancas <iblancasa@gmail.com> Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Goutham <gouthamve@gmail.com> Co-authored-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> Co-authored-by: Chris Marchbanks <csmarchbanks@gmail.com> Co-authored-by: Ben Ye <benye@amazon.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Matthieu MOREL <matthieu.morel35@gmail.com> Co-authored-by: Paulin Todev <paulin.todev@gmail.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Yury Molodov <yurymolodov@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> Co-authored-by: Guillermo Sanchez Gavier <gsanchez@newrelic.com> Co-authored-by: Mikhail Fesenko <proggga@gmail.com> Co-authored-by: Alan Protasio <alanprot@gmail.com>
cstyan
added a commit
that referenced
this pull request
Jul 4, 2024
…support + metadata in WAL) (#14395) * Remote Write 1.1: e2e benchmarks (#13102) * Remote Write e2e benchmarks Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Prometheus ports automatically assigned Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * make dashboard editable + more modular to different job label values Signed-off-by: Callum Styan <callumstyan@gmail.com> * Dashboard improvements * memory stats * diffs look at counter increases Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * run script: absolute path for config templates Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * grafana dashboard improvements * show actual values of metrics * add memory stats and diff Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * dashboard changes Signed-off-by: Callum Styan <callumstyan@gmail.com> --------- Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com> * replace snappy encoding library Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add new proto types Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add decode function for new write request proto Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add lookup table struct that is used to build the symbol table in new write request format Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Implement code paths for new proto format Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * update example server to include handler for new format Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Add new test client Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * tests and new -> original proto mapping util Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add new proto support on receiver end Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Fix test Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * no-brainer copypaste but more performance write support Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove some comented code Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix mocks and fixture Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add basic reduce remote write handler benchmark Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * refactor out common code between write methods Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix: queue manager to include float histograms in new requests Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add sender-side tests and fix failing ones Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * refactor queue manager code to remove some duplication Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix build Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Improve sender benchmarks and some allocations Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Use github.com/golang/snappy Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * cleanup: remove hardcoded fake url for testing Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Add 1.1 version handling code Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Remove config, update proto Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * gofmt Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix NewWriteClient and change new flags wording Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fields rewording in handler Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remote write handler to checks version header Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix typo in log Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * lint Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Add minmized remote write proto format Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add functions for translating between new proto formats symbol table and actual prometheus labels Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add functionality for new minimized remote write request format Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix minor things Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Make LabelSymbols a fixed32 Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove unused proto type Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * update tests Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix build for stringlabels tag Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Use two uint32 to encode (offset,leng) Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * manually optimize varint marshaling Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Use unsafe []byte->string cast to reuse buffer Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix writeRequestMinimizedFixture Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove all code from previous interning approach the 'minimized' version is now the only v1.1 version Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * minimally-tested exemplar support for rw 1.1 Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * refactor new version flag to make it easier to pick a specific format instead of having multiple flags, plus add new formats for testing Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * use exp slices for backwards compat. to go 1.20 plus add copyright header to test file Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix label ranging Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Add bytes slice (instead of slice of 32bit vars) format for testing Co-authored-by: Nicolás Pazos <npazosmendez@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * test additional len and lenbytes formats Co-authored-by: Nicolás Pazos <npazosmendez@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove mistaken package lock changes Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove formats we've decided not to use Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove more format types we probably won't use Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * More cleanup Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * use require instead of assert in custom marshal test Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * cleanup; remove some unused functions Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * more cleanup, mostly linting fixes Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove package-lock.json change again Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * more cleanup, address review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix test panic Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix minor lint issue + use labels Range function since it looks like the tests fail to do `range labels.Labels` on CI Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * new interning format based on []string indeces Co-authored-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove all new rw formats but the []string one also adapt tests to the new format Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * cleanup rwSymbolTable Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * add some TODOs for later Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * don't reserve field 3 for new proto and add TODO Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix custom marshaling Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * lint Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * additional merge fixes Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * lint fixes Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * fix server example Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * revert package-lock.json changes Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * update example prometheus version Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * define separate proto types for remote write 2.0 Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * lint Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * rename new proto types and move to separate pkg Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * update prometheus version for example Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * make proto Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * make Metadata not nullable Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * remove old MinSample proto message Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * change enum names to fit buf build recommend enum naming and lint rules Signed-off-by: Callum Styan <callumstyan@gmail.com> * remote: Added test for classic histogram grouping when sending rw; Fixed queue manager test delay. (#13421) Signed-off-by: bwplotka <bwplotka@gmail.com> * Remote write v2: metadata support in every write request (#13394) * Approach bundling metadata along with samples and exemplars Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Add first test; rebase with main Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Alternative approach: bundle metadata in TimeSeries protobuf Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * update go mod to match main branch Signed-off-by: Callum Styan <callumstyan@gmail.com> * fix after rebase Signed-off-by: Callum Styan <callumstyan@gmail.com> * we're not going to modify the 1.X format anymore Signed-off-by: Callum Styan <callumstyan@gmail.com> * Modify AppendMetadata based on the fact that we be putting metadata into timeseries Signed-off-by: Callum Styan <callumstyan@gmail.com> * Rename enums for remote write versions to something that makes more sense + remove the added `sendMetadata` flag. Signed-off-by: Callum Styan <callumstyan@gmail.com> * rename flag that enables writing of metadata records to the WAL Signed-off-by: Callum Styan <callumstyan@gmail.com> * additional clean up Signed-off-by: Callum Styan <callumstyan@gmail.com> * lint Signed-off-by: Callum Styan <callumstyan@gmail.com> * fix usage of require.Len Signed-off-by: Callum Styan <callumstyan@gmail.com> * some clean up from review comments Signed-off-by: Callum Styan <callumstyan@gmail.com> * more review fixes Signed-off-by: Callum Styan <callumstyan@gmail.com> --------- Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Paschalis Tsilias <paschalist0@gmail.com> * remote write 2.0: sync with `main` branch (#13510) * consoles: exclude iowait and steal from CPU Utilisation 'iowait' and 'steal' indicate specific idle/wait states, which shouldn't be counted into CPU Utilisation. Also see https://github.com/prometheus-operator/kube-prometheus/pull/796 and https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/667. Per the iostat man page: %idle Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request. %iowait Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request. %steal Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor. Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com> * tsdb: shrink txRing with smaller integers 4 billion active transactions ought to be enough for anyone. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * tsdb: create isolation transaction slice on demand When Prometheus restarts it creates every series read in from the WAL, but many of those series will be finished, and never receive any more samples. By defering allocation of the txRing slice to when it is first needed, we save 32 bytes per stale series. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * add cluster variable to Overview dashboard Signed-off-by: Erik Sommer <ersotech@posteo.de> * promql: simplify Native Histogram arithmetics Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com> * Cut 2.49.0-rc.0 (#13270) * Cut 2.49.0-rc.0 Signed-off-by: bwplotka <bwplotka@gmail.com> * Removed the duplicate. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> * Add unit protobuf parser Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * Go on adding protobuf parsing for unit Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * ui: create a reproduction for https://github.com/prometheus/prometheus/issues/13292 Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Get conditional right Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * Get VM Scale Set NIC (#13283) Calling `*armnetwork.InterfacesClient.Get()` doesn't work for Scale Set VM NIC, because these use a different Resource ID format. Use `*armnetwork.InterfacesClient.GetVirtualMachineScaleSetNetworkInterface()` instead. This needs both the scale set name and the instance ID, so add an `InstanceID` field to the `virtualMachine` struct. `InstanceID` is empty for a VM that isn't a ScaleSetVM. Signed-off-by: Daniel Nicholls <daniel.nicholls@resdiary.com> * Cut v2.49.0-rc.1 Signed-off-by: bwplotka <bwplotka@gmail.com> * Delete debugging lines, amend error message for unit Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * Correct order in error message Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * Consider storage.ErrTooOldSample as non-retryable Signed-off-by: Daniel Kerbel <nmdanny@gmail.com> * scrape_test.go: Increase scrape interval in TestScrapeLoopCache to reduce potential flakiness Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Avoid creating string for suffix, consider counters without _total suffix Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> * build(deps): bump github.com/prometheus/client_golang Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.17.0 to 1.18.0. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/client_golang/compare/v1.17.0...v1.18.0) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump actions/setup-node from 3.8.1 to 4.0.1 Bumps [actions/setup-node](https://github.com/actions/setup-node) from 3.8.1 to 4.0.1. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/5e21ff4d9bc1a8cf6de233a3057d20ec6b3fb69d...b39b52d1213e96004bfcb1c61a8a6fa8ab84f3e8) --- updated-dependencies: - dependency-name: actions/setup-node dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * scripts: sort file list in embed directive Otherwise the resulting string depends on find, which afaict depends on the underlying filesystem. A stable file list make it easier to detect UI changes in downstreams that need to track UI assets. Signed-off-by: Jan Fajerski <jfajersk@redhat.com> * Fix DataTableProps['data'] for resultType string Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> * Fix handling of scalar and string in isHeatmapData Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> * build(deps): bump github.com/influxdata/influxdb Bumps [github.com/influxdata/influxdb](https://github.com/influxdata/influxdb) from 1.11.2 to 1.11.4. - [Release notes](https://github.com/influxdata/influxdb/releases) - [Commits](https://github.com/influxdata/influxdb/compare/v1.11.2...v1.11.4) --- updated-dependencies: - dependency-name: github.com/influxdata/influxdb dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.48.0 to 0.48.1. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/prometheus/compare/v0.48.0...v0.48.1) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump client_golang to v1.18.0 (#13373) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Drop old inmemory samples (#13002) * Drop old inmemory samples Co-authored-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Avoid copying timeseries when the feature is disabled Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Run gofmt Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Clarify docs Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Add more logging info Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Remove loggers Signed-off-by: Marc Tuduri <marctc@protonmail.com> * optimize function and add tests Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Simplify filter Signed-off-by: Marc Tuduri <marctc@protonmail.com> * rename var Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Update help info from metrics Signed-off-by: Marc Tuduri <marctc@protonmail.com> * use metrics to keep track of drop elements during buildWriteRequest Signed-off-by: Marc Tuduri <marctc@protonmail.com> * rename var in tests Signed-off-by: Marc Tuduri <marctc@protonmail.com> * pass time.Now as parameter Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Change buildwriterequest during retries Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Revert "Remove loggers" This reverts commit 54f91dfcae20488944162335ab4ad8be459df1ab. Signed-off-by: Marc Tuduri <marctc@protonmail.com> * use log level debug for loggers Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Fix linter Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove noisy debug-level logs; add 'reason' label to drop metrics Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove accidentally committed files Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Propagate logger to buildWriteRequest to log dropped data Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Fix docs comment Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Make drop reason more specific Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Remove unnecessary pass of logger Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Use snake_case for reason label Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Fix dropped samples metric Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> --------- Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> Signed-off-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> * fix(discovery): allow requireUpdate util to timeout in discovery/file/file_test.go. The loop ran indefinitely if the condition isn't met. Before, each iteration created a new timer channel which was always outpaced by the other timer channel with smaller duration. minor detail: There was a memory leak: resources of the ~10 previous timers were constantly kept. With the fix, we may keep the resources of one timer around for defaultWait but this isn't worth the changes to make it right. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Merge pull request #13371 from kevinmingtarja/fix-isHeatmapData ui: fix handling of scalar and string in isHeatmapData * tsdb/{index,compact}: allow using custom postings encoding format (#13242) * tsdb/{index,compact}: allow using custom postings encoding format We would like to experiment with a different postings encoding format in Thanos so in this change I am proposing adding another argument to `NewWriter` which would allow users to change the format if needed. Also, wire the leveled compactor so that it would be possible to change the format there too. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb/compact: use a struct for leveled compactor options As discussed on Slack, let's use a struct for the options in leveled compactor. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb: make changes after Bryan's review - Make changes less intrusive - Turn the postings encoder type into a function - Add NewWriterWithEncoder() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Cut 2.49.0-rc.2 Signed-off-by: bwplotka <bwplotka@gmail.com> * build(deps): bump actions/setup-go from 3.5.0 to 5.0.0 in /scripts (#13362) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3.5.0 to 5.0.0. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/6edd4406fa81c3da01a34fa6f6343087c207a568...0c52d547c9bc32b1aa3301fd7a9cb496313a4491) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github/codeql-action from 2.22.8 to 3.22.12 (#13358) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.22.8 to 3.22.12. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/407ffafae6a767df3e0230c3df91b6443ae8df75...012739e5082ff0c22ca6d6ab32e07c36df03c4a4) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * put @nexucis has a release shepherd (#13383) Signed-off-by: Augustin Husson <augustin.husson@amadeus.com> * Add analyze histograms command to promtool (#12331) Add `query analyze` command to promtool This command analyzes the buckets of classic and native histograms, based on data queried from the Prometheus query API, i.e. it doesn't require direct access to the TSDB files. Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> * included instance in all necessary descriptions Signed-off-by: Erik Sommer <ersotech@posteo.de> * tsdb/compact: fix passing merge func Fixing a very small logical problem I've introduced :(. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb: add enable overlapping compaction This functionality is needed in downstream projects because they have a separate component that does compaction. Upstreaming https://github.com/grafana/mimir-prometheus/blob/7c8e9a2a76fc729e9078889782928b2fdfe240e9/tsdb/compact.go#L323-L325. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Cut 2.49.0 Signed-off-by: bwplotka <bwplotka@gmail.com> * promtool: allow setting multiple matchers to "promtool tsdb dump" command. (#13296) Conditions are ANDed inside the same matcher but matchers are ORed Including unit tests for "promtool tsdb dump". Refactor some matchers scraping utils. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Fixed changelog Signed-off-by: bwplotka <bwplotka@gmail.com> * tsdb/main: wire "EnableOverlappingCompaction" to tsdb.Options (#13398) This added the https://github.com/prometheus/prometheus/pull/13393 "EnableOverlappingCompaction" parameter to the compactor code but not to the tsdb.Options. I forgot about that. Add it to `tsdb.Options` too and set it to `true` in Prometheus. Copy/paste the description from https://github.com/prometheus/prometheus/pull/13393#issuecomment-1891787986 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Issue #13268: fix quality value in accept header Signed-off-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com> * Cut 2.49.1 with scrape q= bugfix. Signed-off-by: bwplotka <bwplotka@gmail.com> * Cut 2.49.1 web package. Signed-off-by: bwplotka <bwplotka@gmail.com> * Restore more efficient version of NewPossibleNonCounterInfo annotation (#13022) Restore more efficient version of NewPossibleNonCounterInfo annotation Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> * Fix regressions introduced by #13242 Signed-off-by: Marco Pracucci <marco@pracucci.com> * fix slice copy in 1.20 (#13389) The slices package is added to the standard library in Go 1.21; we need to import from the exp area to maintain compatibility with Go 1.20. Signed-off-by: tyltr <tylitianrui@126.com> * Docs: Query Basics: link to rate (#10538) Co-authored-by: Julien Pivotto <roidelapluie@o11y.eu> * chore(kubernetes): check preconditions earlier and avoid unnecessary checks or iterations Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Examples: link to `rate` for new users (#10535) * Examples: link to `rate` for new users Signed-off-by: Ted Robertson 10043369+tredondo@users.noreply.github.com Co-authored-by: Bryan Boreham <bjboreham@gmail.com> * promql: use natural sort in sort_by_label and sort_by_label_desc (#13411) These functions are intended for humans, as robots can already sort the results however they please. Humans like things sorted "naturally": * https://blog.codinghorror.com/sorting-for-humans-natural-sort-order/ A similar thing has been done to Grafana, which is also used by humans: * https://github.com/grafana/grafana/pull/78024 * https://github.com/grafana/grafana/pull/78494 Signed-off-by: Ivan Babrou <github@ivan.computer> * TestLabelValuesWithMatchers: Add test case Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * remove obsolete build tag Signed-off-by: tyltr <tylitianrui@126.com> * Upgrade some golang dependencies for resty 2.11 Signed-off-by: Israel Blancas <iblancasa@gmail.com> * Native Histograms: support `native_histogram_min_bucket_factor` in scrape_config (#13222) Native Histograms: support native_histogram_min_bucket_factor in scrape_config --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> * Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram (#13392) Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> * Minor fixes to otlp vendor update script Signed-off-by: Goutham <gouthamve@gmail.com> * build(deps): bump github.com/hetznercloud/hcloud-go/v2 Bumps [github.com/hetznercloud/hcloud-go/v2](https://github.com/hetznercloud/hcloud-go) from 2.4.0 to 2.6.0. - [Release notes](https://github.com/hetznercloud/hcloud-go/releases) - [Changelog](https://github.com/hetznercloud/hcloud-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/hetznercloud/hcloud-go/compare/v2.4.0...v2.6.0) --- updated-dependencies: - dependency-name: github.com/hetznercloud/hcloud-go/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Enhanced visibility for `promtool test rules` with JSON colored formatting (#13342) * Added diff flag for unit test to improvise readability & debugging Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Removed blank spaces Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Fixed linting error Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Added cli flags to documentation Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Revert unrrelated linting fixes Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Fixed review suggestions Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Cleanup Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Updated flag description Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Updated flag description Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> --------- Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * storage: skip merging when no remote storage configured Prometheus is hard-coded to use a fanout storage between TSDB and a remote storage which by default is empty. This change detects the empty storage and skips merging between result sets, which would make `Select()` sort results. Bottom line: we skip a sort unless there really is some remote storage configured. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Remove csmarchbanks from remote write owners (#13432) I have not had the time to keep up with remote write and have no plans to work on it in the near future so I am withdrawing my maintainership of that part of the codebase. I continue to focus on client_python. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com> * add more context cancellation check at evaluation time Signed-off-by: Ben Ye <benye@amazon.com> * Optimize label values with matchers by taking shortcuts (#13426) Don't calculate postings beforehand: we may not need them. If all matchers are for the requested label, we can just filter its values. Also, if there are no values at all, no need to run any kind of logic. Also add more labelValuesWithMatchers benchmarks Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Add automatic memory limit handling Enable automatic detection of memory limits and configure GOMEMLIMIT to match. * Also includes a flag to allow controlling the reserved ratio. Signed-off-by: SuperQ <superq@gmail.com> * Update OSSF badge link (#13433) Provide a more user friendly interface Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> * SD Managers taking over responsibility for registration of debug metrics (#13375) SD Managers take over responsibility for SD metrics registration --------- Signed-off-by: Paulin Todev <paulin.todev@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Björn Rabenstein <github@rabenste.in> * Optimize histogram iterators (#13340) Optimize histogram iterators Histogram iterators allocate new objects in the AtHistogram and AtFloatHistogram methods, which makes calculating rates over long ranges expensive. In #13215 we allowed an existing object to be reused when converting an integer histogram to a float histogram. This commit follows the same idea and allows injecting an existing object in the AtHistogram and AtFloatHistogram methods. When the injected value is nil, iterators allocate new histograms, otherwise they populate and return the injected object. The commit also adds a CopyTo method to Histogram and FloatHistogram which is used in the BufferedIterator to overwrite items in the ring instead of making new copies. Note that a specialized HPoint pool is needed for all of this to work (`matrixSelectorHPool`). --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> * doc: Mark `mad_over_time` as experimental (#13440) We forgot to do that in https://github.com/prometheus/prometheus/pull/13059 Signed-off-by: beorn7 <beorn@grafana.com> * Change metric label for Puppetdb from 'http' to 'puppetdb' Signed-off-by: Paulin Todev <paulin.todev@gmail.com> * mirror metrics.proto change & generate code Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> * TestHeadLabelValuesWithMatchers: Add test case (#13414) Add test case to TestHeadLabelValuesWithMatchers, while fixing a couple of typos in other test cases. Also enclosing some implicit sub-tests in a `t.Run` call to make them explicitly sub-tests. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * update all go dependencies (#13438) Signed-off-by: Augustin Husson <husson.augustin@gmail.com> * build(deps): bump the k8s-io group with 2 updates (#13454) Bumps the k8s-io group with 2 updates: [k8s.io/api](https://github.com/kubernetes/api) and [k8s.io/client-go](https://github.com/kubernetes/client-go). Updates `k8s.io/api` from 0.28.4 to 0.29.1 - [Commits](https://github.com/kubernetes/api/compare/v0.28.4...v0.29.1) Updates `k8s.io/client-go` from 0.28.4 to 0.29.1 - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/kubernetes/client-go/compare/v0.28.4...v0.29.1) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s-io - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s-io ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump the go-opentelemetry-io group with 1 update (#13453) Bumps the go-opentelemetry-io group with 1 update: [go.opentelemetry.io/collector/semconv](https://github.com/open-telemetry/opentelemetry-collector). Updates `go.opentelemetry.io/collector/semconv` from 0.92.0 to 0.93.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-collector/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG-API.md) - [Commits](https://github.com/open-telemetry/opentelemetry-collector/compare/v0.92.0...v0.93.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/collector/semconv dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump actions/upload-artifact from 3.1.3 to 4.0.0 (#13355) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.3 to 4.0.0. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/a8a3f3ad30e3422c9c7b888a15615d19a852ae32...c7d193f32edcb7bfad88892161225aeda64e9392) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump bufbuild/buf-push-action (#13357) Bumps [bufbuild/buf-push-action](https://github.com/bufbuild/buf-push-action) from 342fc4cdcf29115a01cf12a2c6dd6aac68dc51e1 to a654ff18effe4641ebea4a4ce242c49800728459. - [Release notes](https://github.com/bufbuild/buf-push-action/releases) - [Commits](https://github.com/bufbuild/buf-push-action/compare/342fc4cdcf29115a01cf12a2c6dd6aac68dc51e1...a654ff18effe4641ebea4a4ce242c49800728459) --- updated-dependencies: - dependency-name: bufbuild/buf-push-action dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Labels: Add DropMetricName function, used in PromQL (#13446) This function is called very frequently when executing PromQL functions, and we can do it much more efficiently inside Labels. In the common case that `__name__` comes first in the labels, we simply re-point to start at the next label, which is nearly free. `DropMetricName` is now so cheap I removed the cache - benchmarks show everything still goes faster. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * tsdb: simplify internal series delete function (#13261) Lifting an optimisation from Agent code, `seriesHashmap.del` can use the unique series reference, doesn't need to check Labels. Also streamline the logic for deleting from `unique` and `conflicts` maps, and add some comments to help the next person. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * otlptranslator/update-copy.sh: Fix sed command lines Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Rollback k8s.io requirements (#13462) Rollback k8s.io Go modules to v0.28.6 to avoid forcing upgrade of Go to 1.21. This allows us to keep compatibility with the currently supported upstream Go releases. Signed-off-by: SuperQ <superq@gmail.com> * Make update-copy.sh work for both OSX and GNU sed Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Name @beorn7 and @krajorama as maintainers for native histograms I have been the de-facto maintainer for native histograms from the beginning. So let's put this into MAINTAINERS.md. In addition, I hereby proposose George Krajcsovits AKA Krajo as a co-maintainer. He has contributed a lot of native histogram code, but more importantly, he has contributed substantially to reviewing other contributors' native histogram code, up to a point where I was merely rubberstamping the PRs he had already reviewed. I'm confident that he is ready to to be granted commit rights as outlined in the "Maintainers" section of the governance: https://prometheus.io/governance/#maintainers According to the same section of the governance, I will announce the proposed change on the developers mailing list and will give some time for lazy consensus before merging this PR. Signed-off-by: beorn7 <beorn@grafana.com> * ui/fix: correct url handling for stacked graphs (#13460) Signed-off-by: Yury Moladau <yurymolodov@gmail.com> * tsdb: use cheaper Mutex on series Mutex is 8 bytes; RWMutex is 24 bytes and much more complicated. Since `RLock` is only used in two places, `UpdateMetadata` and `Delete`, neither of which are hotspots, we should use the cheaper one. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Fix last_over_time for native histograms The last_over_time retains a histogram sample without making a copy. This sample is now coming from the buffered iterator used for windowing functions, and can be reused for reading subsequent samples as the iterator progresses. I would propose copying the sample in the last_over_time function, similar to how it is done for rate, sum_over_time and others. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Implementation NOTE: Rebased from main after refactor in #13014 Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Add feature flag Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactor concurrency control Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Optimising dependencies/dependents funcs to not produce new slices each request Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Rename flag Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring for performance, and to allow controller to be overridden Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Block until all rules, both sync & async, have completed evaluating Updated & added tests Review feedback nits Return empty map if not indeterminate Use highWatermark to track inflight requests counter Appease the linter Clarify feature flag Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fix typo in CLI flag description Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed auto-generated doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Improve doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Simplify the design to update concurrency controller once the rule evaluation has done Signed-off-by: Marco Pracucci <marco@pracucci.com> * Add more test cases to TestDependenciesEdgeCases Signed-off-by: Marco Pracucci <marco@pracucci.com> * Added more test cases to TestDependenciesEdgeCases Signed-off-by: Marco Pracucci <marco@pracucci.com> * Improved RuleConcurrencyController interface doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Introduced sequentialRuleEvalController Signed-off-by: Marco Pracucci <marco@pracucci.com> * Remove superfluous nil check in Group.metrics Signed-off-by: Marco Pracucci <marco@pracucci.com> * api: Serialize discovered and target labels into JSON directly (#13469) Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> * api: Serialize discovered labels into JSON directly in dropped targets (#13484) Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> * Add ShardedPostings() support to TSDB (#10421) This PR is a reference implementation of the proposal described in #10420. In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing). Follow up work As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes. Signed-off-by: Marco Pracucci <marco@pracucci.com> * storage/remote: document why two benchmarks are skipped One was silently doing nothing; one was doing something but the work didn't go up linearly with iteration count. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Pod status changes not discovered by Kube Endpoints SD (#13337) * fix(discovery/kubernetes/endpoints): react to changes on Pods because some modifications can occur on them without triggering an update on the related Endpoints (The Pod phase changing from Pending to Running e.g.). --------- Signed-off-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: Guillermo Sanchez Gavier <gsanchez@newrelic.com> * Small improvements, add const, remove copypasta (#8106) Signed-off-by: Mikhail Fesenko <proggga@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> * Proposal to improve FPointSlice and HPointSlice allocation. (#13448) * Reusing points slice from previous series when the slice is under utilized * Adding comments on the bench test Signed-off-by: Alan Protasio <alanprot@gmail.com> * lint Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * go mod tidy Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> --------- Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Erik Sommer <ersotech@posteo.de> Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> Signed-off-by: machine424 <ayoubmrini424@gmail.com> Signed-off-by: Daniel Nicholls <daniel.nicholls@resdiary.com> Signed-off-by: Daniel Kerbel <nmdanny@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Jan Fajerski <jfajersk@redhat.com> Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> Signed-off-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: Augustin Husson <augustin.husson@amadeus.com> Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com> Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: tyltr <tylitianrui@126.com> Signed-off-by: Ted Robertson 10043369+tredondo@users.noreply.github.com Signed-off-by: Ivan Babrou <github@ivan.computer> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: Israel Blancas <iblancasa@gmail.com> Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Goutham <gouthamve@gmail.com> Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: SuperQ <superq@gmail.com> Signed-off-by: Ben Kochie <superq@gmail.com> Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Paulin Todev <paulin.todev@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: beorn7 <beorn@grafana.com> Signed-off-by: Augustin Husson <husson.augustin@gmail.com> Signed-off-by: Yury Moladau <yurymolodov@gmail.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> Signed-off-by: Mikhail Fesenko <proggga@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> Co-authored-by: Julian Wiedmann <jwi@linux.ibm.com> Co-authored-by: Bryan Boreham <bjboreham@gmail.com> Co-authored-by: Erik Sommer <ersotech@posteo.de> Co-authored-by: Linas Medziunas <linas.medziunas@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Arianna Vespri <arianna.vespri@yahoo.it> Co-authored-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: daniel-resdiary <109083091+daniel-resdiary@users.noreply.github.com> Co-authored-by: Daniel Kerbel <nmdanny@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jan Fajerski <jfajersk@redhat.com> Co-authored-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: Marc Tudurí <marctc@protonmail.com> Co-authored-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Co-authored-by: Augustin Husson <husson.augustin@gmail.com> Co-authored-by: Björn Rabenstein <beorn@grafana.com> Co-authored-by: zenador <zenador@users.noreply.github.com> Co-authored-by: gotjosh <josue.abreu@gmail.com> Co-authored-by: Ben Kochie <superq@gmail.com> Co-authored-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com> Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: tyltr <tylitianrui@126.com> Co-authored-by: Ted Robertson <10043369+tredondo@users.noreply.github.com> Co-authored-by: Julien Pivotto <roidelapluie@o11y.eu> Co-authored-by: Matthias Loibl <mail@matthiasloibl.com> Co-authored-by: Ivan Babrou <github@ivan.computer> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Israel Blancas <iblancasa@gmail.com> Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Goutham <gouthamve@gmail.com> Co-authored-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> Co-authored-by: Chris Marchbanks <csmarchbanks@gmail.com> Co-authored-by: Ben Ye <benye@amazon.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Matthieu MOREL <matthieu.morel35@gmail.com> Co-authored-by: Paulin Todev <paulin.todev@gmail.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Yury Molodov <yurymolodov@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> Co-authored-by: Guillermo Sanchez Gavier <gsanchez@newrelic.com> Co-authored-by: Mikhail Fesenko <proggga@gmail.com> Co-authored-by: Alan Protasio <alanprot@gmail.com> * remote write 2.0 - follow up improvements (#13478) * move remote write proto version config from a remote storage config to a per remote write configuration option Signed-off-by: Callum Styan <callumstyan@gmail.com> * rename scrape config for metadata, fix 2.0 header var name/value (was 1.1), and more clean up Signed-off-by: Callum Styan <callumstyan@gmail.com> * address review comments, mostly lint fixes Signed-off-by: Callum Styan <callumstyan@gmail.com> * another lint fix Signed-off-by: Callum Styan <callumstyan@gmail.com> * lint imports Signed-off-by: Callum Styan <callumstyan@gmail.com> --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> * go mod tidy Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * Added commmentary to RW 2.0 protocol for easier adoption and explicit semantics. (#13502) * Added commmentary to RW 2.0 protocol for easier adoption and explicit semantics. Signed-off-by: bwplotka <bwplotka@gmail.com> * Apply suggestions from code review Co-authored-by: Nico Pazos <32206519+npazosmendez@users.noreply.github.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Nico Pazos <32206519+npazosmendez@users.noreply.github.com> * prw2.0: Added support for "custom" layouts for native histogram proto (#13558) * prw2.0: Added support for "custom" layouts for native histogram. Result of the discussions: * https://github.com/prometheus/prometheus/issues/13475#issuecomment-1931496924 * https://cloud-native.slack.com/archives/C02KR205UMU/p1707301006347199 Signed-off-by: bwplotka <bwplotka@gmail.com> * prw2.0: Added support for "custom" layouts for native histogram. Result of the discussions: * https://github.com/prometheus/prometheus/issues/13475#issuecomment-1931496924 * https://cloud-native.slack.com/archives/C02KR205UMU/p1707301006347199 Signed-off-by: bwplotka <bwplotka@gmail.com> # Conflicts: # prompb/write/v2/types.pb.go * Update prompb/write/v2/types.proto Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comments, fixed test. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> * first draft of content negotiation Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Lint Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Fix race in test Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Fix another test race Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Almost done with lint Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Fix todos around 405 HEAD handling Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Changes based on review comments Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Update storage/remote/client.go Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Latest updates to review comments Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * latest tweaks Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * remote write 2.0 - content negotiation remediation (#13921) * Consolidate renegotiation error into one, fix tests Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * fix metric name and actuall increment counter Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> --------- Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> * Fixes after main sync. Signed-off-by: bwplotka <bwplotka@gmail.com> * [PRW 2.0] Moved rw2 proto to the full path (both package name and placement) (#13973) undefined * [PRW2.0] Remove benchmark scripts (#13949) See rationales on https://docs.google.com/document/d/1Bpf7mYjrHUhPHkie0qlnZFxzgqf_L32kM8ZOknSdJrU/edit Signed-off-by: bwplotka <bwplotka@gmail.com> * rw20: Update prw commentary after Callum spec review (#14136) * rw20: Update prw commentary after Callum spec review Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Update types.proto Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * [PRW 2.0] Updated spec proto (2.0-rc.1); deterministic v1 interop; to be sympathetic with implementation. (#14330) * [PRW 2.0] Updated spec proto (2.0-rc.1); deterministic v1 interop; to be sympathetic with implementation. Signed-off-by: bwplotka <bwplotka@gmail.com> * update custom marshalling Signed-off-by: bwplotka <bwplotka@gmail.com> * Removed confusing comments. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> * [PRW-2.0] (chain1) New Remote Write 2.0 Config options for 2.0-rc.1 spec. (#14335) NOTE: For simple review this change does not touch remote/ packages, only main and configs. Spec: https://prometheus.io/docs/specs/remote_write_spec_2_0 Supersedes https://github.com/prometheus/prometheus/pull/13968 Signed-off-by: bwplotka <bwplotka@gmail.com> * [PRW-2.0] (part 2) Removed automatic negotiation, updates for the latest spec semantics in remote pkg (#14329) * [PRW-2.0] (part2) Moved to latest basic negotiation & spec semantics. Spec: https://github.com/prometheus/docs/pull/2462 Supersedes https://github.com/prometheus/prometheus/pull/13968 Signed-off-by: bwplotka <bwplotka@gmail.com> # Conflicts: # config/config.go # docs/configuration/configuration.md # storage/remote/queue_manager_test.go # storage/remote/write.go # web/api/v1/api.go * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> * lint Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * storage/remote tests: refactor: extract function newTestQueueManager To reduce repetition. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * use newTestQueueManager for test Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * go mod tidy Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> * [PRW 2.0] (part3) moved type specific conversions to prompb and writev2 codecs. Signed-off-by: bwplotka <bwplotka@gmail.com> * Added test for rwProtoMsgFlagParser; fixed TODO comment. Signed-off-by: bwplotka <bwplotka@gmail.com> * Renamed DecodeV2WriteRequestStr to DecodeWriteV2Request (with tests). Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments on remote_storage example, updated it for 2.0 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed `--enable-feature=metadata-wal-records` docs and error when using PRW 2.0 without it. Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed Callum comments on custom*.go Signed-off-by: bwplotka <bwplotka@gmail.com> * Added TODO to genproto. Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed Callum comments in remote pkg. Signed-off-by: bwplotka <bwplotka@gmail.com> * Added metadata validation to write handler test; fixed ToMetadata. Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed rest of Callum comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed writev2.FromMetadataType (was wrongly using prompb). Signed-off-by: bwplotka <bwplotka@gmail.com> * fix a few import whitespaces Signed-off-by: Callum Styan <callumstyan@gmail.com> * add a default case with an error to the example RW receiver Signed-off-by: Callum Styan <callumstyan@gmail.com> * more minor import whitespace chagnes Signed-off-by: Callum Styan <callumstyan@gmail.com> * Apply suggestions from code review Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Update storage/remote/queue_manager_test.go Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Erik Sommer <ersotech@posteo.de> Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com> Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it> Signed-off-by: machine424 <ayoubmrini424@gmail.com> Signed-off-by: Daniel Nicholls <daniel.nicholls@resdiary.com> Signed-off-by: Daniel Kerbel <nmdanny@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Jan Fajerski <jfajersk@redhat.com> Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> Signed-off-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: Augustin Husson <augustin.husson@amadeus.com> Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com> Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: tyltr <tylitianrui@126.com> Signed-off-by: Ted Robertson 10043369+tredondo@users.noreply.github.com Signed-off-by: Ivan Babrou <github@ivan.computer> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: Israel Blancas <iblancasa@gmail.com> Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Goutham <gouthamve@gmail.com> Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: SuperQ <superq@gmail.com> Signed-off-by: Ben Kochie <superq@gmail.com> Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Paulin Todev <paulin.todev@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: beorn7 <beorn@grafana.com> Signed-off-by: Augustin Husson <husson.augustin@gmail.com> Signed-off-by: Yury Moladau <yurymolodov@gmail.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Leegin <114397475+Leegin-darknight@users.noreply.github.com> Signed-off-by: Mikhail Fesenko <proggga@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> Co-authored-by: Nicolás Pazos <32206519+npazosmendez@users.noreply.github.com> Co-authored-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Nicolás Pazos <npazosmendez@gmail.com> Co-authored-by: alexgreenbank <alex.greenbank@grafana.com> Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Paschalis Tsilias <paschalist0@gmail.com> Co-authored-by: Julian Wiedmann <jwi@linux.ibm.com> Co-authored-by: Bryan Boreham <bjboreham@gmail.com> Co-authored-by: Erik Sommer <ersotech@posteo.de> Co-authored-by: Linas Medziunas <linas.medziunas@gmail.com> Co-authored-by: Arianna Vespri <arianna.vespri@yahoo.it> Co-authored-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: daniel-resdiary <109083091+daniel-resdiary@users.noreply.github.com> Co-authored-by: Daniel Kerbel <nmdanny@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot…
fbegyn
pushed a commit
to fbegyn/prometheus
that referenced
this pull request
Jul 11, 2024
…heus#13398) This added the prometheus#13393 "EnableOverlappingCompaction" parameter to the compactor code but not to the tsdb.Options. I forgot about that. Add it to `tsdb.Options` too and set it to `true` in Prometheus. Copy/paste the description from prometheus#13393 (comment) Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
coleenquadros
added a commit
to stolostron/thanos
that referenced
this pull request
Jul 24, 2024
* CI: increase e2e test parallelism (#6875) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * compact: hook nodownsamplemarkfilter into filters chain (#6893) We have a NoDownsampleMarkFilter that we were not using before in the compactor for some reason. Hook it into the filters chain if downsampling is enabled and then trim matching ULIDs from the downsampling process. Add a test to cover this scenario. Fixes https://github.com/thanos-io/thanos/issues/6179. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * e2e/compact: fix assertion values (#6909) I think the original values were simply too small there - if the runner is slow then it can see smaller metrics values. If the runner works well then the values can go to these bigger values and then the test fails. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * exposes asyncOperationProcessor (#6908) expose struct fix lint Signed-off-by: Ben Ye <benye@amazon.com> * Removed the gitpod from the contribution.md (#6907) Signed-off-by: MeenuyD <meenu.coninja@gmail.com> * .circleci: cache tool deps (#6910) I have noticed that downloading tool deps takes ~5 min in CI each time. We should be able to also cache those tool deps. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * MAINTAINERS.md: add myself as maintainer (#6911) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * blog: Introducing ThanosCon in Paris KubeCon EU 2024 (#6912) * blog: Introducing ThanosCon in Paris KubeCon EU 2024 Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Add image Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * make docs Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Apply suggestions Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Apply suggestions Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Add link to slack channel Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> * Update 2023-20-11-thanoscon.md (typo) (#6915) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added info about ThanosCon to website and Readme. (#6917) * Added info about ThanosCon to website and Readme. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update website/layouts/index.html Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Objstore: Bump Objstore for Azure Workload Identity support (#6891) * Bump objstore for Azure Workload Identity support Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * make check-docs Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * Add changelog entry Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * Update Azure client docs Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * make check-docs Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * Move changelog entry to 0.33.0 release Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * Move changelog entry Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> --------- Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * update compactor backlog doc for checking halt (#6906) Signed-off-by: Ben Ye <benye@amazon.com> * Update LabyrintLabs logo Signed-off-by: Martin Hauskrecht <hauskrechtmartin@gmail.com> * Support reload using signal (#6453) * Support reload using signal Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Add --reloader.method option to sidecar This option allows to tell the sidecar to send a SIGHUP signal to the monitored process to reload its configuration instead of the default HTTP-based method. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Update docs and CHANGELOG.md Signed-off-by: Simon Pasquier <spasquie@redhat.com> --------- Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Support float histogram in store gateway (#6925) * support float histogram in store gateway Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix e2e test Signed-off-by: Ben Ye <benye@amazon.com> * update prompb Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * chore: add truelayer to the adopters (#6933) Signed-off-by: Matteo Martellini <matteo.martellini@truelayer.com> * Fetcher: Add a BlockIDsFetcher Interface to BaseFetcher (#6902) * add BlockIDsFetcher to BaseFetcher Signed-off-by: Wen Xu <wenxuamz@amazon.com> * fix lint Signed-off-by: Wen Xu <wenxuamz@amazon.com> * use chan in the interface method to accept active block ids Signed-off-by: Wen Xu <wenxuamz@amazon.com> * fix comments Signed-off-by: Wen Xu <wenxuamz@amazon.com> * fix lint Signed-off-by: Wen Xu <wenxuamz@amazon.com> * add description of active and parital blocks and modify changelog Signed-off-by: Wen Xu <wenxuamz@amazon.com> * fix interface description Signed-off-by: Wen Xu <wenxuamz@amazon.com> * remove entry in changelog Signed-off-by: Wen Xu <wenxuamz@amazon.com> --------- Signed-off-by: Wen Xu <wenxuamz@amazon.com> * Update hugo file (#6927) * Update hugo file #6889 Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * Version 0.33 removed Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> --------- Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * Updates busybox SHA (#6897) (#6937) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix hotstar logo (#6938) * Fix hotstar logo Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * Delete website/.hugo_build.lock Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> --------- Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> * errutil: fix deadlock Fix deadlock in the Add() function - it accidentally was calling itself recursively. This struct wraps a `MultiError` so we need to call `Add()` on that `MultiError` inside. This deadlock can manifest in Receive - million+ leaking goroutines, all stuck on the MultiTSDB lock. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * update runtime/CloseWithErrCapture comment (#6952) * update runtime/CloseWithErrCapture comment Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com> * fix lint Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com> --------- Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com> * Added a new flag for maximum retention bytes for thanos (#6944) * changes Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> * Changed Int64 to BytesVar Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> * Updated Changelog for new flag in receive Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> --------- Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * fix store gateway span name (#6953) Signed-off-by: Ben Ye <benye@amazon.com> * Upgrade prometheus to latest main (#6950) * upgrade prometheus to latest main Signed-off-by: Ben Ye <benye@amazon.com> * upgrade again Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Improve filtered index cache filtering (#6955) * improve filtered index cache filtering Signed-off-by: Ben Ye <benye@amazon.com> * make inline Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * store: fix misc label_value issues related to external labels (#6879) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * receive: fix adding fields to logger Fix adding fields to the logger because otherwise it looks like "[tenant foo]: (UNDEFINED)". Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Copy labels coming from remote engines (#6957) * Copy labels coming from remote engines When running in distributed mode, the remote engine will use an unsafe cast from ZLabels to Prometheus labels to avoid making new allocations. This makes it hard to use the new gRPC shared buffer pool for receiving and decompressing messages since memory gets retained beyond the scope of a Recv() call. This commit removes the unsafe cast and makes an explicit memory copy of received series labels. Since remote queries are already aggregated series, the amount of data we receive should be small anyway, and the copies on average should have a small impact. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use clone on strings Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add tracing index cache (#6954) * add tracing index cache Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Change order of log for finished compacting blocks (#6966) Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Update MAINTAINERS.md (#6967) Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com> * Add `keep_firing_for` support to alerting rule (#6943) * Add keep_firing_for to alert rules Signed-off-by: JHeil <jheil@coveo.com> * Updating changelog Signed-off-by: JHeil <jheil@coveo.com> * Re-order keepFiringFor at the end of the struct Signed-off-by: JHeil <jheil@coveo.com> --------- Signed-off-by: JHeil <jheil@coveo.com> Signed-off-by: JHeil <52839010+JHeilCoveo@users.noreply.github.com> * Use registerer for gRPC opts and endpoints (#6971) The EndpointSet and StoreGRPC options accept concrete *prometheus.Registry type. This makes it hard to create multiple instances of them because they will try to register the same metrics and the registry will panic. This commit changes the type to a prometheus.Registerer which allows clients to inject a prefixed registry and get metrics for each instance. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * receive: upload compacted blocks if OOO enabled (#6974) * Add streaming series limit at block series client (#6972) * add series limit that is applied when streaming using block series client Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> * add unit tests Signed-off-by: Ben Ye <benye@amazon.com> * address comments Signed-off-by: Ben Ye <benye@amazon.com> * fix comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Updated the defaultChecked property in checkbox element (#6976) Signed-off-by: lpreethvika-samsung <l.preethvika@samsung.com> * Update prometheus to d0c2d9c (#6978) * Update prometheus to d0c2d9c This commit updates Prometheus to commit d0c2d9c which contains the loser-tree based postings merge. Note that we cannot update to latest main since there is currently a bug in PromQL when querying native histograms introduced by https://github.com/prometheus/prometheus/pull/13276. This issue should be fixed by https://github.com/prometheus/prometheus/pull/13289. This commit also updates the Thanos PromQL engine to latest main. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix handler_test.go Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix manager_test.go Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use empty registry for file discovery Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Updates busybox SHA (#6982) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Updates busybox SHA (#6982) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * VERSION: Bump version for release Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * rule: native histogram support (#6390) * Added native histogram support for ruler Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Formatted imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Formated imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed native histogram tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed receiver type Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fix for rebase Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added docs for query endpoints differences Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed comments and naming Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * made HTTPConfig optional Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * made HTTPConfig optional Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted and added check Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixes from comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * renamed queryconfig to clientconfig Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * common prepareEndpointSet Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed lint Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed sidecar Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * mdox ignore checking twitter urls (#7001) Signed-off-by: Ben Ye <benye@amazon.com> * Updated Grofers logo (#7006) Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * TraceID : Fetching TraceID (#6973) * docs: add promcon 2023 thanos talks Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Added website page for companies who offer consultancy and enterprise… (#7000) * Added website page for companies who offer consultancy and enterprise support for Thanos Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * adopters.yml revert Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * retrigger checks Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * added a new line in welcome.md Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * retrigger checks Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> --------- Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * Lazy downloaded index header (#6984) * lazy downloaded index header Signed-off-by: Ben Ye <benye@amazon.com> * update tests Signed-off-by: Ben Ye <benye@amazon.com> * address comments Signed-off-by: Ben Ye <benye@amazon.com> * address comments Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * tests: use remote write in query frontend tests (#6998) * query-frontend: Added support of auto_discovery for memcached (#7004) * query-frontend: Added support of auto_discovery for memcached Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * adjustments to build on main branch Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * CHANGELOG.md Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * typo fixed Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * minor fixex after review Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> --------- Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * Fix URI encoding of strings Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * ui: enable partial response strategy by default Rebuild Signed-off-by: Vanshika <vanshikav928@gmail.com> * Added negative offset check for caching queries (#7011) Signed-off-by: pawarpranav83 <pawarpranav@gmail.com> Co-authored-by: pawarpranav83 <pawarpranav@gmail.com> * Update MAINTAINERS.md (Bartek's company) Should be done in January last year... Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * store: add chunksize tests to acceptance tests * add chunk size tests to acceptance tests * refactor acceptance tests slightly Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * added tasrie it services as support partner (#7023) * added tasrie it services as support partner Signed-off-by: Tasrie IT Services <info@tasrieit.com> * added the link in the list.html page Signed-off-by: Tasrie IT Services <info@tasrieit.com> * fixed the logo resolution to 190x70 Signed-off-by: Tasrie IT Services <info@tasrieit.com> * Remove extraneous text Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Tasrie IT Services <info@tasrieit.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Query Frontend: Add tenant label to metrics (#6887) This commit adds a tenant label to the HTTP metrics which are exported by the Query Frontend. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * tests: use remote write in query frontend tests (#7017) * resolved issue #7024 Signed-off-by: Pratham Agarwal <agarwalpratham1812@gmail.com> * QueryFrontend|Query: Create new arg to enable extended functions (#7028) * Adding new parameter for extended functions in querier Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com> * Adding new flag for QFE Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * improve argument passing in query side Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding changelog Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding e2e test for query Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * undoing uneeded changes Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing docs Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing e2e tests Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixing backward compat test Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * changes from CR Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com> Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * receive: respect forward timeout in http handler buckets (#7030) Forward timeout is what ultimately decides how long a HTTP request might take so it doesn't make sense to hard-code max value of 5. In this pull request I propose respecting the configured forward timeout and adding extra buckets if the default buckets don't cover it completely. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * receive: do not leak grpc connections (#7031) * receive: do not leak grpc connections Prevent a leak in gRPC connections by garbage collecting old ones when the hashring changes. For that purpose, I propose adding a `Nodes() string` method so that it would be possible to know what nodes do not exist in the hashring anymore. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * receive: change order of operations Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * fix multi-tenancy link (#7032) Signed-off-by: Alec Rajeev <ccarajeev342@gmail.com> * removing todo comments from query docs Signed-off-by: Harsh Pratap Singh <harshpratapsingh8210@gmail.com> * Query: add optional tenancy enforcement (#6756) * Query: add optional tenancy enforcement With this commit it's now possible to enable enforcement of tenancy. If tenancy is enabled, a tenant label will be added to queries based on the tenant information provided by the tenant header, and the tenant-label-name. The implementation for query APIs are done by using prom-label-proxy as library, while the implementation for non-query APIs are written from scratch. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Add changelog entry Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Query: Add non-default tenant testcase Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Test: make query a constant to make linter happy Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Address review comments - Remove empty lines - If multiple tenant matchers are found in the original query, we only replace the first one with the header provided tenant, and remove any subsequent ones. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Address review comments - `--enable-tenancy` -> `--enforce-tenancy` - Create `RewritePromQL` and `RewriteLabelMatchers` to clean up code in query api. Also move getLabelMatchers to tenancy pkg. - Use prom-label-proxys `EnforceMatchers` to rewrite labels on non-query APIs instead of own solution - Don't specifically handle `illegalLabelMatcherError` Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Re-arrage go.mod to make linter happy. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Address review comments Minor changes to CLI docs, code-comments and changelog. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Query: Add tenancy docs This commit adds documentation for the tenancy features. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Update docs/components/query.md Review comment Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Jacob Baungård Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> Signed-off-by: Jacob Baungård Hansen <jacobbaungard@redhat.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Tests: unique docker env name for tenant tests (#7038) The e2e tests would occasionally fail due to non-unqiue docker environment names. With this commit the tests are environments are given unique names to avoid these failures. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * changed default 3m to 15m Signed-off-by: Player256 <dattucodes@gmail.com> * Fix docs (#7039) Fix docs post #6539 merge. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Revert "receive: upload compacted blocks if OOO enabled (#6974)" (#7053) This reverts commit 7b8eb86c0ff3b1144aa1cea392806afa40a4cdf8. Proper way to handle this is to disable vertical compaction. I am trying to add this functionality here: https://github.com/prometheus/prometheus/pull/13393 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * [CHORE] adding thanos upload-snapshot command (#6884) Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com> * fix docs error Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> * cmd: make bucket upload command take lset from flags (#7059) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * CHANGELOG: mark 0.34 as in progress Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * VERSION: cut release 0.34.0-rc.0 Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * UI: Don't always force tracing (#7062) Forced tracing was.. Forced true always, even if the checkbox in the UI to enable tracing was not actually checked. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * go.mod: update Prometheus version (#7047) Update Prometheus version to include https://github.com/prometheus/prometheus/pull/13242 which is important for me - it unblocks further postings work. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Make RetryError and HaltError able to be fetched for root cause (#7043) * Make RetryError and HaltError able to be fetched for root cause Signed-off-by: Alex Le <leqiyue@amazon.com> * Added unit test Signed-off-by: Alex Le <leqiyue@amazon.com> * fix lint Signed-off-by: Alex Le <leqiyue@amazon.com> * fixed IsRetryError and IsHaltError functions Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> * receive: disable overlapping compaction Use the new TSDB flag to disable overlapping compaction to fix OOO samples handling in the Receive component. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CI: Ensure static react-app is checked in (#7063) * CI: Ensure static react-app is checked in With this commit the CI system should fail if changes to the react-app has been made without checking in the changes. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Add `react-app` as dependency `check-react-app` To ensure the react-app is rebuilt before checking for changes. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Receive: refactor handler for improved readability and organization (#6898) * [wip] First checkpoint Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [wip] Second checkpoint All tests passing, unit and e2e. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Small random refactors Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add some useful trace tags Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Concurrent and traced local writes Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve variable names in remote writes Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rename `newFanoutForward` function Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * More refactors Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix linting issue Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a quorum test with sloppy quorum Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [wip] Try to make retries work Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [wip] Checkpoint: wait group still hanging Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Some refactors Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add some commented code so I don't lose it Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Adapt tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove sloppy quorum code Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Move some code around Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove even more leftover of sloppy quorum Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract a type to hold function params Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove unused struct field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove useless variable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove type that wasn't used enough Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Delete function to tighten up max buffered responses Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add comments to some functions Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix peer up check Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix size of replication tracking slices Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rename context Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Don't do local writes concurrently Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove extra error logging Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix syntax after merge Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add missing methods to peersContainer Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix handler test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reset peers state on hashring changes Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Handle PR comment regarding waitgroup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Set span tags to help debug Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix concurrency issue We close the request as soon as quorum is reached and leave a few Go routines running to finish replication and so cleanups. This means that the context from the HTTP request is cancelled... which ends up also cancelling the pending replication requests. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix request ID middleware Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix `distributeTimeseriesToReplicas` comment Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract var with 1-indexed replication index Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rename methods in peersContainer interface Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make peerGroup `getConnection` check if peers are up Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove yet one more not useful log Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove logger from `h.sendWrites` Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Upgrade grpc to 1.57.2 (#7078) 1、In the replace of go.mod, due to weaveworks/common#239, The grpc version is 1.45.0, but there are vulnerabilities in this version. In order to fix CVE-2023-44478, the grpc version needs to be upgraded to 1.57.2 2、In order to upgrade GRPC, the version of weaveworks/common also needs to be upgraded, otherwise the build will fail Signed-off-by: hanyuting8 <hytxidian@163.com> * Store: acceptance test for proxy store (#7084) * Add basic acceptance tests for proxy store * Fix bug where invalid requests got ignored because of partial response strategy Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Fix lazy postings with zero length (#7083) * fix lazy postings with zero length Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> * unit tests Signed-off-by: Ben Ye <benye@amazon.com> * fix doc Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Store: fix label values edge case (#7082) If the requested label is an external label and we have series matchers we should only return results if the series matchers actually match a series. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * VERSION: cut release 0.34.0-rc.1 Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * receive: race condition in handler Close() when stopped early (#7087) Receiver hangs waiting for the HTTP Hander to shutdown if an error occurs before Handler is initialized. This might happen, for example, if the hashring is too small for a given replication factor. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> * receive: use async remote writing (#7045) * CHANGELOG: cut release 0.34 (#7095) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Stores: convert tests to not rely on slice labels (#7098) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * all: get rid of query pushdown to simplify query path (#7014) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: dont rely on slice labels continued Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Update prometheus/prometheus (#7096) * Update prometheus/prometheus This commit updates prometheus/prometheus to latest main (60b6266e). Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix file discovery Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * receive/handler: fix locking twice (#7112) Fix bug introduced in https://github.com/thanos-io/thanos/pull/6898: we were RLock()ing twice. This leads to a deadlock in some situations. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * fix minio store gateway err (#7114) Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * receive/handler: do not double lock (#7124) markPeerUnavailable was always taking a lock and in one case we were calling it with a lock already taken. Fix this. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * default to alertmanager v2 api (#7123) Signed-off-by: Jake Keeys <jake@keeys.org> * Receive: dont rely on slice labels (#7100) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Upgrade grpc to 1.57.2 (#7078) 1、In the replace of go.mod, due to weaveworks/common#239, The grpc version is 1.45.0, but there are vulnerabilities in this version. In order to fix CVE-2023-44478, the grpc version needs to be upgraded to 1.57.2 2、In order to upgrade GRPC, the version of weaveworks/common also needs to be upgraded, otherwise the build will fail Signed-off-by: hanyuting8 <hytxidian@163.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * VERSION: cut release 0.34.1 Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * docs: update helm installation instruction the prometheus helm chart is a community maintained chart since a few years. With that, the old example pointed to an old chart and the provided example values aren't also working anymore. This update the documentation. Signed-off-by: Mario Constanti <github@constanti.de> * docs: fix link (#7129) The link has moved to another since Cisco bought Banzai Cloud. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: run make docs for helm installation instruction Signed-off-by: Mario Constanti <github@constanti.de> * Fixing log line for remote engine in debug mode (#7133) Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding new method on BucketedBytes to expose used memory (#7137) * Adding new method on bucketed bytes to expose used Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Removing interface, using RWMutex Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding InfraCloud as Enterprise support partner (#7141) * adding InfraCloud as Enterprise support partner Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io> * replaced svg file to match layout Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io> * added alt-text and horizontal image Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io> --------- Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io> * docs: fix link (#7129) The link has moved to another since Cisco bought Banzai Cloud. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * cache: attach object storage hash to iter key (#6880) Attach object storage hash to the iter key so that it would be possible to reuse the same cache storage e.g. Redis for different buckets. Without this, the results are funny to say the least if you accidentally attempt to do that. Thus, let's add the hash to reduce the possibility of an accident for our users. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Add support for extended promql functions in rule (#7105) Adds a flag to register the extended promql functions supported by the thanos query engine when running the rule component. This will allow rule config files containing query expressions with (xrate / xincrease / xdelta) to pass validation. This will only work if the query endpoint in use is running the thanos engine. Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com> * Extended func support - doc update (#7161) * Add support for extended promql functions in rule Adds a flag to register the extended promql functions supported by the thanos query engine when running the rule component. This will allow rule config files containing query expressions with (xrate / xincrease / xdelta) to pass validation. This will only work if the query endpoint in use is running the thanos engine. Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com> * Update rendered docs with added flag Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com> --------- Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com> * Copy labels from remote instant queries (#7151) Similar to https://github.com/thanos-io/thanos/pull/6957, we should copy labels from remote instant queries so that memory does not get overwritten when processing series in a central engine. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * e2e/query_frontend: add tests for explain/analyze (#7160) Adding tests for explain/analyze with QFE. Will add fixes as separate PR. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cache/caching_bucket: add path to hash (#7158) Add path to the hash. This allows identifying difference instances by different config paths. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Allow using different listing strategies (#7134) * Allow using different listing strategies Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Expose flags for block list strategy Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Run make docs Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix whitespace Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add CHANGELOG entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cache: implement the circuit breaker pattern for asynchronous set operations in the cache client (#7010) * Implement the circuit breaker pattern for asynchronous set operations in the cache client Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Add feature flag for circuitbreaker Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Sync docs Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Skip configuration validation if the circuit breaker is disabled Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Make lint happy Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Abstract the logic of the circuit breaker Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> --------- Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * queryfrontend: fix analysis after API changes Fix the analysis functionality with query-frontend after the recent changes. Added tests for this. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * bugfix: lazy posting optimization with wrong cardinality for estimation (#7122) * bugfix: catch lazy posting optimization using wrong cardinality for estimation Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * *: properly treat native histogram deduplication in chunk series merger We have detected a problem in the chunk seriers merger where it will panic in case it encounters native histogram chunks. I am using thanos as a library for a project and wanted to use the penalty function to dedup blocks from Prometheus instances. Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Query UI: Add tenant box (#6867) * Query UI: Add tenant box With this commit as tenant box is added to the query UI. It can be used to specify which tenant to use when making a query. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Re-compile static react app Recompiles the static react app as now needed following: https://github.com/thanos-io/thanos/pull/6900 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Move changelog item to appropiate future release After merging it was under the 0.34 release. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Move query path tenancy proposal to done Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * receive: add support for globbing tenant specifiers (#7155) We want to be able to route all tenants which begin with certain letters to some receivers so we need to have some kind of globbing/regex support in the hashring. This PR adds that functionality. We've been using this in prod successfully. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * .circleci: bump setup_docker_version version (#7177) The current image is deprecated. See https://discuss.circleci.com/t/remote-docker-image-deprecations-and-eol-for-2024/50176. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Unhide distributed execution mode This commit exposes the distributed query execution mode to end-users by unhiding the flag used to toggle this feature. The commit also adds documentation on when the mode is appropriate to be used. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add changelog entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cortex/querier: fix analysis merging (#7179) We were not merging analysis properly - mergo was overwriting data. Instead of using a whole library for this, just write two small functions and use them. Add test to cover this. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fix docs Fixes docs formatting and updates the distributed execution link to the done proposal. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Added platformengineers Signed-off-by: Munir Khakhi <munir@improwised.com> * Added platformengineers (#7181) Signed-off-by: Munir Khakhi <munir@improwised.com> * Bump google.golang.org/protobuf to v1.33.0 (#7191) This PR bumps the version of google.golang.org/protobuf to v1.33.0 fix a potential vulnerability in the protojson.Unmarhsl function [1] that can occure when unmarshaling a message with a protobuf value. Even if the function isn't used directly in Thanos it would be safer to just bump it directly. [1] https://pkg.go.dev/vuln/GO-2024-2611 Signed-off-by: Daniel Mellado <dmellado@redhat.com> * fix: make responsive support page and change size of platform-engineer logo fix: add anchor tag to all images Signed-off-by: Payal17122000 <raviyapayal17@gmail.com> * downsample: retry objstore related errors Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * downsample: retry objstore related errors Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * rule: do not turn off if resolving fails (#7192) Do not turn off Ruler if resolving fails. We can still (try to) evaluate rules even if Alertmanager is not available. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Query UI: Only show tenant box with enforcement on (#7186) With this commit we only show the tenant-ui box when enforcement of tenancy is on, as it is not needed otherwise. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Reloader: Add support for watching and decompressing Prometheus configuration directories (#7199) Signed-off-by: Daniel Hrabovcak <thespiritxiii@gmail.com> * [FEAT] Adding blog post (#7202) Signed-off-by: Helia Barroso <helia.barroso@hotmail.com> Co-authored-by: Helia Barroso <helia.barroso@hotmail.com> * storepb: make ServerAsClient channels unbuffered Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add support for TSDB selector in querier (#7200) * Add support for TSDB selector in querier This PR allows using the query distributed mode against a set of multi-tenant receivers as described in https://github.com/thanos-io/thanos/blob/main/docs/proposals-done/202301-distributed-query-execution.md#distributed-execution-against-receive-components. The feature is enabled by a selector.relabel-config flag in the Query component which allows it to select a subset of TSDBs to query based on their external labels. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add CHANGELOG entry and fix docs Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix tests Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add comments Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for MatchersForLabelSets Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix failing test Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use an unbuffered channel Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Change flag description Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Remove parameter from ServerAsClient Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Update thanos-io/promql-engine (#7215) * Update thanos-io/promql-engine This commit updates the promql-engine module to latest main and modifies to remote engine based on the breaking change. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix lint Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * expose NewPromSeriesSet (#7214) Signed-off-by: Ben Ye <benye@amazon.com> * [CHORE] adding auto GOMEMLIMIT flag (#7223) Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * bump objstore package version to latest main (#7228) Signed-off-by: Ben Ye <benye@amazon.com> * Query-frontend: fix missing redis username config (#7224) * add username cfg to rueidis client Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * UI: Showing Block Size Stats (#7233) * feat(ui): added BlockSizeStats calculation to blocks page A block can have a list of contained files set in `.thanos.files`. If the `files` array is set, all referenced files with `size_bytes` set are counted: - sum of all `chunk/*` file sizes - size of index file - total size (sum of both) Shows statistics about the selected block in the block details view: - Total size of block - Size of index (and percentage of total) - Size of all chunks (and percentage of total) - Daily growth, based on total size and block duration Output is humanized up to Pebibytes and fixed to two decimal places; raw bytes are accessible through mouse over / title text. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * feat(ui): added aggregated BlockSizeStats to blocks row title Added total size of all blocks from a source to the row title, beneath the source name. The shown total size is humanized up to pebibytes and fixed to two decimal places; raw bytes value is accessible through mouse over / title text. The shown value will refresh with selected compaction levels, but doesn't take block filter into account. I thought about showing daily growth as well, but just summing all milliseconds of all blocks doesn't work with overlapping blocks / multiple resolutions. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * chore(docs): added UI block size PR to CHANGELOG.md Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * chore(ui): removed comments Automatic code formatting duplicated some comments near import statements. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> --------- Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * Fix lazy expanded postings cache and bug of non equal matcher (#7220) * fix lazy expanded postings cache and bug of non equal matcher with non existent values Signed-off-by: Ben Ye <benye@amazon.com> * test case for remove keys noop Signed-off-by: Ben Ye <benye@amazon.com> * add promqlsmith fuzz test Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix go mod Signed-off-by: Ben Ye <benye@amazon.com> * rename test Signed-off-by: Ben Ye <benye@amazon.com> * fix series request timestamp Signed-off-by: Ben Ye <benye@amazon.com> * skip e2e test Signed-off-by: Ben Ye <benye@amazon.com> * handle non lazy expanded case Signed-off-by: Ben Ye <benye@amazon.com> * update comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Bump Prometheus to include new label regex optimization (#7232) * bump Prometheus version to include new label matcher regex value optimization Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * fix again Signed-off-by: Ben Ye <benye@amazon.com> * include latest fix Signed-off-by: Ben Ye <benye@amazon.com> * update go mod Signed-off-by: Ben Ye <benye@amazon.com> * fix explain test Signed-off-by: Ben Ye <benye@amazon.com> * fix test again Signed-off-by: Ben Ye <benye@amazon.com> * update again Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * fix tests so far Signed-off-by: Ben Ye <benye@amazon.com> * fix compactor tests Signed-off-by: Ben Ye <benye@amazon.com> * use own out of order chunk index Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * update docs for receive routing only with limits (#7241) Signed-off-by: Alec Rajeev <13004609+alecrajeev@users.noreply.github.com> * docs: add thanoscon 2024 talks (#7243) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * remove write method from Compactor interface (#7246) Signed-off-by: Ben Ye <benye@amazon.com> * change shipper to not overwrite all external labels (#7247) Signed-off-by: Ben Ye <benye@amazon.com> * fix(changelog): fix GOMEMLIMIT pull request reference Signed-off-by: roth-wine <philipp.roth@hetzner.com> * Update kakkoyun's affiliation (#7251) * Added Shield in adopters (#7254) * Added Shield in adopters Signed-off-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com> * Upload compatible image Signed-off-by: suhas-chikkanna <162577490+suhas-chikkanna@users.noreply.github.com> --------- Signed-off-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com> Signed-off-by: suhas-chikkanna <162577490+suhas-chikkanna@users.noreply.github.com> Co-authored-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com> * Tracing: added missing sampler types (#7231) * added missing sampler types Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> * added changelog entry Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> * fixed changelog entry Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> * Fixed changelog entry conflict Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> --------- Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> * fix query_test when --race enabled (#7258) Signed-off-by: Yi Jin <yi.jin@databricks.com> * go.mod: bump promql-engine (#7263) Bump promql-engine version to include samples counting. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fix 7244 error targets page (#7245) * added UNKNOWN to TargetHealth_value at targets proto Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com> * added TargetHealth_value UNKNOWN to rpc.pb.go Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com> --------- Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com> * clarify documentation around selecor.relabel-config option Signed-off-by: notafile <nota@notafile.com> * clarify writing around sharding Signed-off-by: notafile <nota@notafile.com> * don't halt compaction due to overlapping sources when vertical compaction is enabled (#7225) Signed-off-by: Ben Ye <benye@amazon.com> * Propagate the query plan * Serialize the plan for remote executions latest engine Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Propagate marshaled plan and introduce optimizer Propagating the query plan in the remote engine requests and introduce new SetProjectionColumns optimizer Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixing passing down of plan Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * go mod tidy Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * avoid panics Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * delete dev file Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * undo small refactor Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * improve test Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> generating protos Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> fixing v1 Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> delete unused method Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Set projection labels after distributing queries * removing second precision engine, upstream already truncates Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Passing the plan along as the query in remote executions Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Using proper constructors passing the query plan Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Refactor query creation from plan Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fallback in case we cant use plan Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * refactor, add tests Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Refactor to method Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * last nits Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * api/ui: show peak/total samples in analyze (#7269) Show the new peak/total fields in analyze output next to each operator. Add tooltips to explain what is the meaning of each field. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * receive/handler: implement tenant label splitting (#7256) * receive/handler: implement tenant label splitting Implement splitting incoming HTTP requests along some label inside of the timeseries themselves. This functionality is useful when you have one big application exposing lots of series and, for instance, you have a label `team` that identifies different owners of metrics in that application. Then using this you can use that `team` label to have different tenants in Thanos. Only negative thing that I could spot is that if after splitting one of the requests fails then that code is used for all tenants and that skews the Receiver metrics a little bit. I think that can be left as a TODO task. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * test/e2e: add more receiver tests Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * thanos/receive: note that splitting takes precendence over HTTP Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * thanos/receive: fix typo Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * query: fixing dedup iterator when working on mixed sample types (#7271) * query: fixing dedup iterator when working on mixed sample types There was a panic in case the dedupiterator worked on two chunks with both Native Histograms and Float (XOR encoded). Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding changelog Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing lint Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * removing comments Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixing repro test case Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing initialization Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing changelog Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * adding header to new file Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * using t.Run Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing ordering of samples in tests Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * change the reflect package to an unsafe package (#7143) - as 'reflect.String.Header' is deprecated, it is replaced with an unsafe package. Signed-off-by: Youngjun <yj.yoo@okestro.com> * Receive: fix issue-7248 with parallel receive_forward (#7267) * Receive: fix issue-7248 by introducing a worker pool Signed-off-by: Yi Jin <yi.jin@databricks.com> * fix unit test bug Signed-off-by: Yi Jin <yi.jin@databricks.com> * fix CLI flags not pass into the receive handler Signed-off-by: Yi Jin <yi.jin@databricks.com> * address comments Signed-off-by: Yi Jin <yi.jin@databricks.com> * init context in constructor Signed-off-by: Yi Jin <yi.jin@databricks.com> --------- Signed-off-by: Yi Jin <yi.jin@databricks.com> * Fix incorrect comments (#7268) Signed-off-by: Magiceses <magiceses0118@gmail.com> * [CHORE] adding user agent (#7281) Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com> * receive/multitsdb: do not delete not uploaded blocks (#7166) * receive/multitsdb: do not delete not uploaded blocks If a block hasn't been uploaded yet then tell the TSDB layer not to delete them. This prevents a nasty race where the TSDB layer can delete a block before the shipper gets to it. I saw this happen with a very small block. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * receive/multitsdb: change order Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * shipper/receive: just use a single lock Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Compact: Replace group with resolution in ownsample metrics (#7283) Compaction dowsnample metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a resolution label to reduce cardinality. Fixes: #5841 Signed-off-by: SuperQ <superq@gmail.com> * Propagate warnings from instant queries Warnings from remote instant queries get turned into errors, which is a bug. It should be up to the root client to decide whether warnings should be show as such, or converted to errors. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add CHANGELOG entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Receive: stop relying on grpc server config to set grpc client secure/skipVerify Signed-off-by: Guillaume Lecerf <guillaume.lecerf@iziwork.com> * Show warnings in query frontend (#7289) * Show warnings in query frontend QFE currently does not parse warnings from downstream queriers. This commit fixes that by adding the field to proto messages and modifies the merge function to take warnings into account. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add CHANGELOG entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Omit empty warnings Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Optimize empty posting check in lazy posting (#7298) * change lazy postings empty posting check to use cardinality Signed-off-by: Ben Ye <benye@amazon.com> * update lazy posting test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Sidecar: mark as unqueryable if prometheus is down (#7297) If the prometheus that belongs to a sidecar is down we dont need to query the sidecar. This PR makes it so that we take the sidecar out of the endpoint set then. We do the same for all other store APIs by retuning an error in the info/Info gRPC call if they are marked as not ready. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * [CHORE] considering X-Forwarded-For on HTTP Logging (#7303) Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com> * Query|Receiver: Do not log full request on ProxyStore by default (#7305) * Query|Receiver|Store: Do not log full request on ProxyStore by default We had a problem on our production where a sudden increase in requests with long matchers was putting our receivers under a lot of pressure. Upon checking pro…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This functionality is needed in downstream projects because they have a separate component that does compaction.
Upstreaming
https://github.com/grafana/mimir-prometheus/blob/7c8e9a2a76fc729e9078889782928b2fdfe240e9/tsdb/compact.go#L323-L325.