TestManagedRegistry_externalLabels is flaky #2881
Description
The TestManagedRegistry_externalLabels test is flaky, and will randomly fail.
➜ tempo git:(main) go test ./modules/generator/registry --run TestManagedRegistry_externalLabels -v --count=1
=== RUN TestManagedRegistry_externalLabels
--- PASS: TestManagedRegistry_externalLabels (0.00s)
PASS
ok github.com/grafana/tempo/modules/generator/registry 0.338s
➜ tempo git:(main) go test ./modules/generator/registry --run TestManagedRegistry_externalLabels -v --count=1
=== RUN TestManagedRegistry_externalLabels
registry_test.go:252:
Error Trace: /Users/lasse.hels/Projects/tempo/modules/generator/registry/registry_test.go:252
/Users/lasse.hels/Projects/tempo/modules/generator/registry/registry_test.go:150
Error: elements differ
extra elements in list A:
([]interface {}) (len=2) {
(registry.sample) {
l: (labels.Labels) (len=3) {
(labels.Label) {
Name: (string) (len=5) "__foo",
Value: (string) (len=3) "bar"
},
(labels.Label) {
Name: (string) (len=22) "__metrics_gen_instance",
Value: (string) (len=13) "MMDKF7QLWWFW0"
},
(labels.Label) {
Name: (string) (len=8) "__name__",
Value: (string) (len=10) "my_counter"
}
},
t: (int64) 1693557980698,
v: (float64) 0
},
(registry.sample) {
l: (labels.Labels) (len=3) {
(labels.Label) {
Name: (string) (len=5) "__foo",
Value: (string) (len=3) "bar"
},
(labels.Label) {
Name: (string) (len=22) "__metrics_gen_instance",
Value: (string) (len=13) "MMDKF7QLWWFW0"
},
(labels.Label) {
Name: (string) (len=8) "__name__",
Value: (string) (len=10) "my_counter"
}
},
t: (int64) 1693557980698,
v: (float64) 1
}
}
extra elements in list B:
([]interface {}) (len=2) {
(registry.sample) {
l: (labels.Labels) (len=3) {
(labels.Label) {
Name: (string) (len=22) "__metrics_gen_instance",
Value: (string) (len=13) "MMDKF7QLWWFW0"
},
(labels.Label) {
Name: (string) (len=5) "__foo",
Value: (string) (len=3) "bar"
},
(labels.Label) {
Name: (string) (len=8) "__name__",
Value: (string) (len=10) "my_counter"
}
},
t: (int64) 1693557980698,
v: (float64) 0
},
(registry.sample) {
l: (labels.Labels) (len=3) {
(labels.Label) {
Name: (string) (len=22) "__metrics_gen_instance",
Value: (string) (len=13) "MMDKF7QLWWFW0"
},
(labels.Label) {
Name: (string) (len=5) "__foo",
Value: (string) (len=3) "bar"
},
(labels.Label) {
Name: (string) (len=8) "__name__",
Value: (string) (len=10) "my_counter"
}
},
t: (int64) 1693557980698,
v: (float64) 1
}
}
listA:
([]registry.sample) (len=2) {
(registry.sample) {
l: (labels.Labels) (len=3) {
(labels.Label) {
Name: (string) (len=5) "__foo",
Value: (string) (len=3) "bar"
},
(labels.Label) {
Name: (string) (len=22) "__metrics_gen_instance",
Value: (string) (len=13) "MMDKF7QLWWFW0"
},
(labels.Label) {
Name: (string) (len=8) "__name__",
Value: (string) (len=10) "my_counter"
}
},
t: (int64) 1693557980698,
v: (float64) 0
},
(registry.sample) {
l: (labels.Labels) (len=3) {
(labels.Label) {
Name: (string) (len=5) "__foo",
Value: (string) (len=3) "bar"
},
(labels.Label) {
Name: (string) (len=22) "__metrics_gen_instance",
Value: (string) (len=13) "MMDKF7QLWWFW0"
},
(labels.Label) {
Name: (string) (len=8) "__name__",
Value: (string) (len=10) "my_counter"
}
},
t: (int64) 1693557980698,
v: (float64) 1
}
}
listB:
([]registry.sample) (len=2) {
(registry.sample) {
l: (labels.Labels) (len=3) {
(labels.Label) {
Name: (string) (len=22) "__metrics_gen_instance",
Value: (string) (len=13) "MMDKF7QLWWFW0"
},
(labels.Label) {
Name: (string) (len=5) "__foo",
Value: (string) (len=3) "bar"
},
(labels.Label) {
Name: (string) (len=8) "__name__",
Value: (string) (len=10) "my_counter"
}
},
t: (int64) 1693557980698,
v: (float64) 0
},
(registry.sample) {
l: (labels.Labels) (len=3) {
(labels.Label) {
Name: (string) (len=22) "__metrics_gen_instance",
Value: (string) (len=13) "MMDKF7QLWWFW0"
},
(labels.Label) {
Name: (string) (len=5) "__foo",
Value: (string) (len=3) "bar"
},
(labels.Label) {
Name: (string) (len=8) "__name__",
Value: (string) (len=10) "my_counter"
}
},
t: (int64) 1693557980698,
v: (float64) 1
}
}
Test: TestManagedRegistry_externalLabels
--- FAIL: TestManagedRegistry_externalLabels (0.00s)
FAIL
FAIL github.com/grafana/tempo/modules/generator/registry 0.346s
FAIL
➜ tempo git:(main) go test ./modules/generator/registry --run TestManagedRegistry_externalLabels -v --count=1
=== RUN TestManagedRegistry_externalLabels
--- PASS: TestManagedRegistry_externalLabels (0.00s)
PASS
ok github.com/grafana/tempo/modules/generator/registry 0.346sLooks like the label ordering might be the root cause. Other tests may also be affected, as the collectRegistryMetricsAndAssert function is used in several tests in registry_test.go.
Tested on commit 13bbd12.