3.0 migration guide #15099

jan--f · 2024-10-04T14:41:38Z

Just a first outline of the 3.0 migration guide. Looking forward to feedback!

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

tjhop · 2024-10-06T23:05:07Z

docs/stability.md

+    function](https://github.com/prometheus/prometheus/issues/2458)
+  * Remote remote read and the remote read endpoint
+* Server-side HTTPS and basic authentication
+* Service discovery integrations, with the exception of `static_configs` and `file_sd_configs`


Should HTTP SD be included in this list now as well?

tjhop · 2024-10-06T23:08:10Z

docs/migration.md

+- Some previously deprecated feature flags have been removed. Check that you no 
+  longer pass any of the following to `--enable-feature`: `promql-at-modifier`
+  `promql-negative-offset`, `remote-write-receiver`, 
+  `new-service-discovery-manager`, `expand-external-labels`, 
+  `no-default-scrape-port`.


I think this would be easier to follow with the options as a proper list

Yes. And state that you can still pass in these flags, just that they don't do anything anymore, and that a warning will be logged.

roidelapluie · 2024-10-07T08:54:44Z

We should remove references to 2.0 and link to the 2.55 docs to migrate from prom1 to prom2. STABILITY should only contain 3.0 stuff.

we should mention that it is only possible to rollback to 2.55 because of indexing change.

beorn7

Great start.

Left some comments and ideas.

beorn7 · 2024-10-08T14:48:21Z

docs/migration.md

+
+## PromQL
+
+- Regular expressions in PromQL not match newline characters.


not → now

But maybe also explain this better. IIRC this is about . that did not match newline before but now does, while you could of course create a regexp matching the newline character even in the old world (just not by using .).

As this is a migration guide, we should maybe provide a few examples (old regexp now also matches this other string, here is what you have to do to keep the old behavior).

beorn7 · 2024-10-08T14:51:11Z

docs/migration.md

+- Regular expressions in PromQL not match newline characters.
+- Lookback and range selectors are left open and right closed (previously left 
+  closed and right closed). This mostly means that the result size of range 
+  selectors is more predictable.


We need to explain more details, like that this is happening with "perfect" alignment, most typically happening in tests, so you probably have to change a lot of tests, but not a lot in practice.

With the special caveat of sub queries. As some frontends create nicely aligned queries (like Grafana), and sub-queries are internally always aligned to multiples of their step size, you will get this a lot with sub queries.

beorn7 · 2024-10-08T14:53:27Z

docs/migration.md

+  closed and right closed). This mostly means that the result size of range 
+  selectors is more predictable.
+- The `holt_winters` function has been renamed to `double_exponential_smoothing` 
+  and is now guarded by the `experimental_promql` feature flag.


The flag is called promql-experimental-functions.

We need to say things more clearly, along these lines:

If you want to keep using holt_winters, you have to do both of these things:

Rename holt_winters to double_exponential_smoothing in your query.

Make sure to pass --enable-feature=promql-experimental-functions.

(Wordsmithing still required.)

beorn7 · 2024-10-08T14:54:24Z

docs/migration.md

+- Some previously deprecated feature flags have been removed. Check that you no 
+  longer pass any of the following to `--enable-feature`: `promql-at-modifier`
+  `promql-negative-offset`, `remote-write-receiver`, 
+  `new-service-discovery-manager`, `expand-external-labels`, 
+  `no-default-scrape-port`.


Yes. And state that you can still pass in these flags, just that they don't do anything anymore, and that a warning will be logged.

beorn7 · 2024-10-08T14:55:26Z

docs/migration.md

+  `new-service-discovery-manager`, `expand-external-labels`, 
+  `no-default-scrape-port`.
+
+## PromQL


I think each point in the list below needs to be an own paragraph (or even subsection with heading) with examples and concrete migration steps (see comments below for inspiration).

docs/migration.md

beorn7 · 2024-10-08T15:11:46Z

Going through Prometheus 3.0 (view) should give more inspiration what to add here (and looking at the changelog, and making sure that all is consistent with each other).

Some things I have spotted:

UTF-8 support in name is now always on (might be surprising for some that special characters might show up in names now).
The whole "we won't interfere with you scrape ports anymore" thing, i.e. if people still want to see the default HTTP or HTTPS port, they have to specify it in their scrape config (and vice versa: if they don't want to see it, they must not specify it).

docs/migration.md

beorn7 · 2024-10-09T11:10:22Z

#15092 should be mentioned, too, i.e. the whole "selectors are actually expected to return matching results". This will only affect a few, but because it is so subtle, I would still call it out. Mostly for people who use "remote read hacks" (i.e. the whole "let's put a MySQL query in a label selector", maybe even with a hint how to still make the hack work (by adding a corresponding label to the result)).

jan--f · 2024-10-09T12:58:14Z

Thanks everyone so far. I'll address the feedback later this week, until then please keep it coming.

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

jan--f · 2024-10-11T14:05:48Z

Here is a first update according to reviews. There are some formatting warts to fix..

I'm still unsure about the "regex match \n" examples. its tricky to strike the right balance between accurate and succinct.

Happt to incorporate feedback via PRs against https://github.com/jan--f/prometheus/tree/3.0-migration too.

jan--f · 2024-10-11T14:06:19Z

Oh and I still want to add links to issues and PRs.

docs/stability.md

docs/migration.md

ywwg · 2024-10-16T14:39:43Z

docs/migration.md

+### UTF-8 names
+Prometheus v3 supports UTF-8 in metrics and label names. This means metrics and 
+label names can change after upgrading according to what is exposed by 
+endpoints.


I need to expand this to explain how to adjust configs to disallow utf8. Should I do that here or in a subsequent PR?

You can either create a PR against https://github.com/jan--f/prometheus/tree/3.0-migration or we merge this soon and then incremental improvements can be parallelized in p/p.

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

Signed-off-by: Owen Williams <owen.williams@grafana.com>

bwplotka · 2024-10-18T07:33:54Z

docs/migration.md

-This document offers guidance on migrating from Prometheus 1.8 to Prometheus 2.0 and newer versions.
+In line with our [stability promise](https://prometheus.io/docs/prometheus/latest/stability/),
+the Prometheus 3.0 release contains a number of backwards incompatible changes.
+This document offers guidance on migrating from Prometheus 2.x to Prometheus 3.0 and newer versions.

 ## Flags


Can we add # Configuration

#15178 got renamed. While native histograms were experimental, it's worth mentioning the rename?

docs/migration.md

docs/migration: details on enabling legacy name validation

docs/migration.md

beorn7 · 2024-10-23T22:04:19Z

The change to log/slog should be mentioned as it changes the logging format.

old:

ts=2024-10-23T22:01:06.074Z caller=main.go:627 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2024-10-23T22:01:06.074Z caller=main.go:671 level=info msg="Starting Prometheus Server" mode=server version="(version=, branch=, revision=91d80252c3e528728b0f88d254dd720f6be07cb8-modified)"
ts=2024-10-23T22:01:06.074Z caller=main.go:676 level=info build_context="(go=go1.23.0, platform=linux/amd64, user=, date=, tags=unknown)"
ts=2024-10-23T22:01:06.074Z caller=main.go:677 level=info host_details="(Linux 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 gigafips (none))"

new:

time=2024-10-24T00:03:07.542+02:00 level=INFO source=/home/beorn/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:640 msg="No time or size retention was set so using the default time retention" duration=15d
time=2024-10-24T00:03:07.542+02:00 level=INFO source=/home/beorn/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:681 msg="Starting Prometheus Server" mode=server version="(version=, branch=, revision=7c7116fea8343795cae6da42960cacd0207a2af8)"
time=2024-10-24T00:03:07.542+02:00 level=INFO source=/home/beorn/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:686 msg="operational information" build_context="(go=go1.23.0, platform=linux/amd64, user=, date=, tags=unknown)" host_details="(Linux 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 gigafips (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"

beorn7 · 2024-10-23T22:05:31Z

Scrape config option scrape_classic_histograms is now called always_scrape_classic_histograms.

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

jan--f · 2024-10-24T11:44:43Z

Ok I addressed all recent comments. I would like to merge the current state if there are no glaring errors so its included in the RC release due at the end of this week.

Further additions and fixes can be added after that.

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

docs/migration.md

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

jesusvazquez

LGTM!

jan--f added 2 commits October 4, 2024 16:39

docs: 2 to 3 migration guide

15c5235

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

docs/stability: add 3.0 section

73cd2b4

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

tjhop reviewed Oct 6, 2024

View reviewed changes

beorn7 reviewed Oct 8, 2024

View reviewed changes

ArthurSens reviewed Oct 8, 2024

View reviewed changes

docs/migration.md Outdated Show resolved Hide resolved

docs/migration: address first review round

6a12eee

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

jan--f force-pushed the 3.0-migration branch from f680fce to 6a12eee Compare October 11, 2024 14:02

alexgreenbank reviewed Oct 16, 2024

View reviewed changes

docs/stability.md Outdated Show resolved Hide resolved

docs/migration.md Show resolved Hide resolved

ywwg mentioned this pull request Oct 16, 2024

Epic: Implement UTF-8 character support for label and metric names #13095

Closed

40 tasks

ywwg reviewed Oct 16, 2024

View reviewed changes

alexgreenbank mentioned this pull request Oct 17, 2024

scrape: provide a fallback format #15136

Merged

2 tasks

jan--f and others added 2 commits October 17, 2024 16:36

Second review round

56d2dbc

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

docs/migration: details on enabling legacy name validation

bf2e5e4

Signed-off-by: Owen Williams <owen.williams@grafana.com>

bwplotka reviewed Oct 18, 2024

View reviewed changes

ArthurSens reviewed Oct 18, 2024

View reviewed changes

docs/migration.md Show resolved Hide resolved

Merge pull request #5 from ywwg/owilliams/3.0-migration

960c0bf

docs/migration: details on enabling legacy name validation

krajorama reviewed Oct 22, 2024

View reviewed changes

docs/migration.md Show resolved Hide resolved

jan--f added 2 commits October 24, 2024 10:23

migration: address more comments

188ab26

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

migration: add log format and le normalization

7fb0b57

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

migration: add new enable_http2 default for remote write

342dca3

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

jan--f enabled auto-merge October 25, 2024 09:40

GiedriusS reviewed Oct 25, 2024

View reviewed changes

docs/migration.md Outdated Show resolved Hide resolved

migration: fix typo

d8050d6

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

jan--f disabled auto-merge October 25, 2024 09:48

jan--f enabled auto-merge (squash) October 25, 2024 09:49

jesusvazquez approved these changes Oct 25, 2024

View reviewed changes

jan--f merged commit f131cdd into prometheus:main Oct 25, 2024
26 checks passed

julienduchesne pushed a commit to julienduchesne/prometheus that referenced this pull request Dec 13, 2024

3.0 migration guide (prometheus#15099)

a1bcab5


		## PromQL

		- Regular expressions in PromQL not match newline characters.

3.0 migration guide #15099

3.0 migration guide #15099

Uh oh!

Conversation

jan--f commented Oct 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roidelapluie commented Oct 7, 2024

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

beorn7 commented Oct 8, 2024

Uh oh!

Uh oh!

beorn7 commented Oct 9, 2024

Uh oh!

jan--f commented Oct 9, 2024

Uh oh!

jan--f commented Oct 11, 2024

Uh oh!

jan--f commented Oct 11, 2024

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

beorn7 commented Oct 23, 2024

Uh oh!

beorn7 commented Oct 23, 2024

Uh oh!

jan--f commented Oct 24, 2024

Uh oh!

Uh oh!

jesusvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!