Add UTF-8 support in metric and label names #689

fedetorres93 · 2024-09-12T15:47:55Z

As part of the initiative for adding UTF-8 support to Prometheus, this PR brings UTF-8 support for metric and label names as described in this proposal (for example: {"metric.name", "label.name"="value"}.

For UTF-8 label names defined in the URL path, an escaping syntax is used. For example, the label "label.name"="value" would be escaped like: /metrics/job/example/U__label_2e_name/value. This escaping is compatible with the base64 encoding for label values: /metrics/job/example/U__label_2e_name@base64/dmFsdWU=

UTF-8 support can be enabled by running the Pushgateway with the flag --push.enable-utf8-names=true

Addresses #623

Signed-off-by: Federico Torres <federico.torres@grafana.com>

beorn7 · 2024-09-18T16:23:11Z

Sorry, still fighting my backlog. I hope I get to review this tomorrow.

beorn7

Thank you for doing this.
A few ideas about how to precisely control whether to escape or not. I hope I got it right, but happy to learn where I'm wrong.

In different news, this needs documentation (which currently is just in the ever growing README.md – feel free to add it there, breaking down the documentation into "proper" documentation files would be a task for another PR).

beorn7 · 2024-09-20T13:18:49Z

handler/handler_test.go

@@ -18,6 +18,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"github.com/prometheus/common/model"


This has to go into the next block (line 29).

beorn7 · 2024-09-20T13:23:49Z

main.go

@@ -17,6 +17,7 @@ import (
 	"compress/gzip"
 	"context"
 	"fmt"
+	"github.com/prometheus/common/model"


As above, has to go to line 39 or so.

beorn7 · 2024-09-20T13:30:42Z

main.go

@@ -73,6 +74,7 @@ func main() {
 		persistenceFile     = app.Flag("persistence.file", "File to persist metrics. If empty, metrics are only kept in memory.").Default("").String()
 		persistenceInterval = app.Flag("persistence.interval", "The minimum interval at which to write out the persistence file.").Default("5m").Duration()
 		pushUnchecked       = app.Flag("push.disable-consistency-check", "Do not check consistency of pushed metrics. DANGEROUS.").Default("false").Bool()
+		pushEscaped         = app.Flag("push.enable-escaped-labels", "Allow escaped characters in label names in URLs.").Default("false").Bool()


Let's call this push.enable-utf8-names to mirror the feature flag in Prometheus, and also to correctly describe what we are doing. (Without the flag, the PGW will reject non-legacy characters in names even in the payload of a push.)
Variable name and doc string has to be adjusted accordingly.

main.go

beorn7 · 2024-09-20T13:39:51Z

handler/push.go

@@ -180,19 +180,21 @@ func splitLabels(labels string) (map[string]string, error) {
 	for i := 0; i < len(components)-1; i += 2 {
 		name, value := components[i], components[i+1]
 		trimmedName := strings.TrimSuffix(name, Base64Suffix)
+		unescapedName := model.UnescapeName(trimmedName, model.ValueEncodingEscaping)


We don't even want to try to unescape if the corresponding flag is not set. To not refer to the flag value directly here, maybe set a package-level variable of type model.EscapingScheme to either model.ValueEncodingEscaping or model.NoEscaping and then use that variable here.

beorn7 · 2024-09-20T13:40:18Z

handler/push.go

 		if !model.LabelNameRE.MatchString(trimmedName) ||
+			!model.LabelName(unescapedName).IsValid() ||


With this validity test, wo don't need the one in the previous line anymore, do wo?

Signed-off-by: Federico Torres <federico.torres@grafana.com>

fedetorres93 · 2024-09-23T21:52:33Z

@beorn7 Thanks for your feedback, the requested changes were addressed.

beorn7

Code looks good (modulu doc comment on push.EscapingScheme. But we also need documentation. (As suggested previously, I believe it is OK for now to add a section to the ever growing README.md, keeping in mind that eventually we would like to break it up into "proper" documentation.)

beorn7 · 2024-09-25T16:16:18Z

handler/push.go

@@ -41,6 +41,8 @@ const (
 	Base64Suffix = "@base64"
 )

+var EscapingScheme = model.NoEscaping


This could benefit from a doc comment.

Signed-off-by: Federico Torres <federico.torres@grafana.com>

fedetorres93 · 2024-09-25T21:36:18Z

@beorn7 Sorry about the documentation, thanks for pointing it out, now it's added. Also added EscapingScheme comment.

beorn7

Cool. I just couldn't resist and applied a lot of wordsmithing. Please see my suggestions.

beorn7 · 2024-09-26T16:53:21Z

README.md

@@ -320,6 +320,28 @@ Examples:

      /metrics/job/titan/name@base64/zqDPgc6_zrzOt864zrXPjc-C

+### UTF-8 support


Let's be a bit more specific here and call this "UTF-8 support for metric and label names".
(UTF-8 is supported for label values already.)

beorn7 · 2024-09-26T17:21:10Z

README.md

+UTF-8 characters in metric and label names are supported by enabling
+the `--push.enable-utf8-names` flag. The syntax is as follows:
+
+    {"some.metric", "foo.bar"="baz"}
+
+For UTF-8 label names defined in the URL path, [an escaping syntax](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping) is used.


I would not explain the syntax of the text exposition format here. (It also doesn't cover protobuf.)

With some wordsmithing included, I would propose the following:

Suggested change

UTF-8 characters in metric and label names are supported by enabling

the `--push.enable-utf8-names` flag. The syntax is as follows:

{"some.metric", "foo.bar"="baz"}

For UTF-8 label names defined in the URL path, [an escaping syntax](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping) is used.

Newer versions of the Prometheus exposition formats (text and protobuf)

support the full UTF-8 character set in metric and label names. The

Pushgateway only accepts special characters in names if the command line

flag `--push.enable-utf8-names` is set.

To allow special characters in label names that are part of the URL path, the flag also enables a

[specific encoding mechanism](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping). This is similar to the base64 encoding for label _values_ described above,

but works differently in detail for technical and historical reasons. As before, client libraries

should usually take care of the encoding. It works as follows:

beorn7 · 2024-09-26T17:40:36Z

README.md

+* Prefix the string with `U__`.
+* All non-valid characters (i.e. characters other than letters, numbers, and underscores) will be encoded as underscores surrounding the Unicode value, like `_1F60A_`.
+* All pre-existing underscores will become doubled: `__`.
+* If a string should start with "U__" already, it will need to be escaped: `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).


Let's stay consistent with regard to the form. You start with an imperative form and then use future tense.

Here my suggestion with a few things added:

Suggested change

* Prefix the string with `U__`.

* All non-valid characters (i.e. characters other than letters, numbers, and underscores) will be encoded as underscores surrounding the Unicode value, like `_1F60A_`.

* All pre-existing underscores will become doubled: `__`.

* If a string should start with "U__" already, it will need to be escaped: `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).

* A label name containing encoded characters starts with `U__`.

* All non-standard characters (i.e. characters other than letters, numbers, and underscores) are encoded as underscores surrounding their Unicode value, like `_1F60A_`.

* All pre-existing underscores are encoded as a double-underscore: `__`.

* If a label name starts with `U__` already, these characters have to be encoded as well, resulting in `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).

* A label name starting with `U__` in it's encoded form, but containing invalid sequences (e.g. `U__in_xxx_valid`) is left unchanged.

beorn7 · 2024-09-26T17:41:24Z

README.md

+* All pre-existing underscores will become doubled: `__`.
+* If a string should start with "U__" already, it will need to be escaped: `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).
+
+For example, the label `"foo.bar"="baz"` would be escaped like: 


Let's better use "encoded" rather than "escaped" to only use one word throughout this file.

beorn7 · 2024-09-26T17:41:44Z

README.md

+
+    /metrics/job/example/U__foo_2e_bar/baz
+
+This escaping is compatible with the base64 encoding for label values:


escaping → encoding, see above.

beorn7 · 2024-09-26T17:42:30Z

README.md

+This escaping is compatible with the base64 encoding for label values:
+
+    /metrics/job/example/U__foo_2e_bar@base64/YmF6
+


Let's add something about the one edge case, e.g.

Note that this method has an unlikely edge case that is not handled properly: A pusher unaware of the encoding mechanism might use a label name that is also a valid encoded version of another label name. E.g. if a pusher intends to use the label name U__foo_2e_bar, but doesn't encode it as U___55_____foo__2e__bar, the Pushgateway will decode U__foo_2e_bar to foo.bar. This is the main reason why the decoding is opt-in via the --push.enable-utf8-names flag.

Signed-off-by: Federico Torres <federico.torres@grafana.com>

fedetorres93 · 2024-09-27T14:47:04Z

@beorn7 Thanks for the docs suggestions, done.

beorn7

🚢

Adds UTF-8 support for metric and label names. These changes are based on the work done on the Prometheus common libraries [here](prometheus/common#537) and [here](prometheus/common#570) - The `prometheus-metrics-exposition-formats` module will use the new quoting syntax `{"foo"}` iff the metric does not conform to the legacy name format (`foo{}`) - The `prometheus-metrics-model` module has a new flag (`nameValidationScheme`) that determines if validation is done using the legacy or the UTF-8 scheme. This flag can be set via a property in the properties file. - Scrapers can announce via content negotiation that they support UTF-8 names by adding `escaping=allow-utf-8` in the Accept header. In cases where UTF-8 is not available, metric providers can be configured to escape names in a few different ways: values (`U__` UTF value escaping for perfect round-tripping), underscores (all invalid chars become `_`), dots (dots become `_dot_`, `_` becomes `__`, all other values become `___`). Escaping has a global default (`PrometheusNaming.DEFAULT_ESCAPING_SCHEME`) or can also be specified in Accept header with the `escaping=` term, which can be `allow-utf-8` (for UTF-8-compatible), `underscores`, `dots`, or `values`. This should still be a noop for existing configurations because scrapers will not be passing the escaping key in the Accept header. Existing functionality is maintained. - The `prometheus-metrics-exporter-pushgateway` module will [escape](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping) UTF-8 grouping keys in the URL path used when pushing metrics (see prometheus/pushgateway#689) Work towards prometheus/prometheus#13095 --------- Signed-off-by: Federico Torres <federico.torres@grafana.com> Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com> Co-authored-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

fedetorres93 added 3 commits September 4, 2024 10:56

Allow UTF-8 escaped characters in label names in URLs

81d3593

Signed-off-by: Federico Torres <federico.torres@grafana.com>

Add unit tests

519870f

Signed-off-by: Federico Torres <federico.torres@grafana.com>

Merge remote-tracking branch 'upstream/master' into ftorres/utf-8

83c45eb

fedetorres93 marked this pull request as ready for review September 12, 2024 16:01

beorn7 self-assigned this Sep 17, 2024

beorn7 requested changes Sep 20, 2024

View reviewed changes

fedetorres93 force-pushed the ftorres/utf-8 branch 2 times, most recently from ab2490d to 83c45eb Compare September 23, 2024 21:48

Change UTF-8 flag name and validations

90f4e80

Signed-off-by: Federico Torres <federico.torres@grafana.com>

fedetorres93 requested a review from beorn7 September 23, 2024 21:51

beorn7 requested changes Sep 25, 2024

View reviewed changes

Add documentation

1c39f43

Signed-off-by: Federico Torres <federico.torres@grafana.com>

fedetorres93 requested a review from beorn7 September 25, 2024 21:33

beorn7 requested changes Sep 26, 2024

View reviewed changes

Documentation changes

62d5b8a

Signed-off-by: Federico Torres <federico.torres@grafana.com>

fedetorres93 requested a review from beorn7 September 27, 2024 14:47

beorn7 approved these changes Oct 2, 2024

View reviewed changes

beorn7 merged commit c408b8e into prometheus:master Oct 2, 2024
4 checks passed

ywwg mentioned this pull request Nov 13, 2024

Add support for utf8 names on /v1/label/:name/values endpoint prometheus/prometheus#14943

Closed

fedetorres93 mentioned this pull request Dec 12, 2024

Fix UTF-8 label names display in UI #710

Merged

fedetorres93 mentioned this pull request Jan 9, 2025

UTF-8 support in metric and label names prometheus/client_java#1255

Merged

gigermocas mentioned this pull request Feb 4, 2025

Update pushgateway from 1.9.0 to 1.11.0 gigermocas/prometheus-rpm#6

Merged

		if !model.LabelNameRE.MatchString(trimmedName) \|\|
		!model.LabelName(unescapedName).IsValid() \|\|

		@@ -320,6 +320,28 @@ Examples:

		/metrics/job/titan/name@base64/zqDPgc6_zrzOt864zrXPjc-C

		### UTF-8 support

-UTF-8 characters in metric and label names are supported by enabling
-the `--push.enable-utf8-names` flag. The syntax is as follows:
-    {"some.metric", "foo.bar"="baz"}
-For UTF-8 label names defined in the URL path, [an escaping syntax](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping) is used.
+Newer versions of the Prometheus exposition formats (text and protobuf)
+support the full UTF-8 character set in metric and label names. The
+Pushgateway only accepts special characters in names if the command line
+flag `--push.enable-utf8-names` is set.
+To allow special characters in label names that are part of the URL path, the flag also enables a
+[specific encoding mechanism](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping). This is similar to the base64 encoding for label _values_ described above,
+but works differently in detail for technical and historical reasons. As before, client libraries
+should usually take care of the encoding. It works as follows:

-* Prefix the string with `U__`.
-* All non-valid characters (i.e. characters other than letters, numbers, and underscores) will be encoded as underscores surrounding the Unicode value, like `_1F60A_`.
-* All pre-existing underscores will become doubled: `__`.
-* If a string should start with "U__" already, it will need to be escaped: `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).
+* A label name containing encoded characters starts with `U__`.
+* All non-standard characters (i.e. characters other than letters, numbers, and underscores) are encoded as underscores surrounding their Unicode value, like `_1F60A_`.
+* All pre-existing underscores are encoded as a double-underscore: `__`.
+* If a label name starts with `U__` already, these characters have to be encoded as well, resulting in `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).
+* A label name starting with `U__` in it's encoded form, but containing invalid sequences (e.g. `U__in_xxx_valid`) is left unchanged.


		/metrics/job/example/U__foo_2e_bar/baz

		This escaping is compatible with the base64 encoding for label values:

		This escaping is compatible with the base64 encoding for label values:

		/metrics/job/example/U__foo_2e_bar@base64/YmF6

Add UTF-8 support in metric and label names #689

Add UTF-8 support in metric and label names #689

Uh oh!

Conversation

fedetorres93 commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beorn7 commented Sep 18, 2024

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fedetorres93 commented Sep 23, 2024

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fedetorres93 commented Sep 25, 2024

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fedetorres93 commented Sep 27, 2024

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fedetorres93 commented Sep 12, 2024 •

edited

Loading