[GEP-19] Migrate seed Prometheus deployment and configuration #9180

rfranzke · 2024-02-16T16:41:27Z

How to categorize this PR?

/area dev-productivity monitoring
/kind enhancement

What this PR does / why we need it:
Similar to #9128, this PR migrates management of the seed Prometheus deployment and its configuration to prometheus-operator.

As part of this, I adapted the VerticalPodAutoscaler spec for Prometheis in the pkg/componen/tmonitoring/prometheus package to reflect the changes from #9175.

Which issue(s) this PR fixes:
Part of #9065

Special notes for your reviewer:
/cc @oliver-goetz @ScheererJ
FYI @istvanballok @rickardsjp

Release note:

It is now possible to provide configuration for the seed Prometheus running in seed clusters' `garden` namespaces. Read all about it [here](https://github.com/gardener/gardener/tree/master/docs/extensions/logging-and-monitoring.md#seed-prometheus).

ScheererJ

Thanks a lot for this nicely structured improvement.

pkg/component/kubestatemetrics/resources.go

pkg/component/resourcemanager/resource_manager.go

pkg/component/monitoring/charts/bootstrap/charts/monitoring/templates/config.yaml

pkg/component/vpa/recommender.go

pkg/component/monitoring/prometheus/seed/podmonitors.go

pkg/component/monitoring/charts/bootstrap/charts/monitoring/templates/config.yaml

pkg/component/monitoring/prometheus/seed/podmonitors.go

ScheererJ

Thanks a lot for the explanations.

/lgtm

gardener-prow · 2024-02-20T09:01:42Z

LGTM label has been added.

Git tree hash: 1b19ef1289dd91a58938c9b91d7a148d83fc7361

oliver-goetz · 2024-02-20T09:53:38Z

/assign

oliver-goetz

Nice PR, thanks 🚀

pkg/component/monitoring/prometheus/seed/scrapeconfigs.go

pkg/component/monitoring/prometheus/seed/podmonitors.go

oliver-goetz · 2024-02-20T14:19:20Z

pkg/component/monitoring/prometheus/vpa.go

@@ -43,11 +44,15 @@ func (p *prometheus) vpa() *vpaautoscalingv1.VerticalPodAutoscaler {
 			},
 			ResourcePolicy: &vpaautoscalingv1.PodResourcePolicy{
 				ContainerPolicies: []vpaautoscalingv1.ContainerResourcePolicy{
+					{
+						ContainerName:    "*",


Does VPA merge ContainerPolicies? Otherwise, there won't be a match. Afaik prometheus runs 2 container only (prometheus, config-reloader) which have already dedicated policies.
I suggest to include this setting into prometheus container, config-reloader is not scaled anyway.

Interesting, we just talked about this a few hours ago with @voelzmo 😄 Actually, we don't know for sure. I was just reflecting the change from #9175 so that it also gets applied when Prometheus is managed via prometheus-operator. In the scope of this PR, this is correct/what we should do. If we think this VPA definition is wrong, then let's fix it separately (my goal is just to translate/refactor the Helm charts to "mgmt via prom-operator", not to change/add/remove/reinvent things). :) OK?

I'm pretty sure now, that VPA does not merge ContainerPolicy (ref).
Anyway, we can fix it in an own PR too.

Thanks for checking @oliver-goetz!
@voelzmo @nickytd Would you like to open a PR that addresses these concerns?

done: #9192

rfranzke · 2024-02-20T15:37:28Z

I added a new commit which adapts the seed-care controller to incorporate the health of the new prometheus-seed ManagedResource when computing the seed conditions (similar to #9189) :)

oliver-goetz · 2024-02-20T16:17:51Z

/lgtm
/approve

gardener-prow · 2024-02-20T16:17:55Z

LGTM label has been added.

Git tree hash: 9dd09ae768678b31d23323dd896e0bd9837d3390

gardener-prow · 2024-02-20T16:17:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: oliver-goetz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [oliver-goetz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

We don't use a ServiceMonitor here because the `kubernetes_sd_configs` uses `role=service` which is only supported via `ScrapeConfig` resources.

…tor`

oliver-goetz · 2024-02-21T08:10:33Z

/lgtm
🚀

gardener-prow · 2024-02-21T08:10:38Z

LGTM label has been added.

Git tree hash: e2e301c6b81b4932a1b893c3b8424fc7cd409a02

gardener-prow bot requested review from oliver-goetz and ScheererJ February 16, 2024 16:41

rfranzke mentioned this pull request Feb 16, 2024

[GEP-19] Migrate monitoring stack to prometheus-operator #9065

Closed

46 tasks

gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 16, 2024

rfranzke force-pushed the gep19/seed-prometheus branch from 6e6f01c to e3f5d16 Compare February 19, 2024 08:31

gardener-prow bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Feb 19, 2024

ScheererJ reviewed Feb 19, 2024

View reviewed changes

rfranzke force-pushed the gep19/seed-prometheus branch from e3f5d16 to 013d45e Compare February 19, 2024 12:26

gardener-prow bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 19, 2024

rfranzke force-pushed the gep19/seed-prometheus branch from 013d45e to 40c4510 Compare February 19, 2024 12:28

rfranzke requested a review from ScheererJ February 19, 2024 12:28

rfranzke force-pushed the gep19/seed-prometheus branch 2 times, most recently from 6dbd32c to b99221f Compare February 19, 2024 16:08

ScheererJ reviewed Feb 20, 2024

View reviewed changes

gardener-prow bot assigned ScheererJ Feb 20, 2024

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Feb 20, 2024

gardener-prow bot assigned oliver-goetz Feb 20, 2024

oliver-goetz reviewed Feb 20, 2024

View reviewed changes

rfranzke force-pushed the gep19/seed-prometheus branch from b99221f to 1bc2821 Compare February 20, 2024 15:35

gardener-prow bot removed the lgtm Indicates that a PR is ready to be merged. label Feb 20, 2024

gardener-prow bot requested a review from oliver-goetz February 20, 2024 15:35

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Feb 20, 2024

gardener-prow bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Feb 20, 2024

rfranzke added 12 commits February 21, 2024 09:06

Translate kube-state-metrics scrape config into ScrapeConfig

27f8cce

We don't use a ServiceMonitor here because the `kubernetes_sd_configs` uses `role=service` which is only supported via `ScrapeConfig` resources.

Translate gardener-resource-manager scrape config into `ServiceMoni…

96dc50d

…tor`

Translate vpa-recommender scrape config into PodMonitor

c8766e8

Translate prometheus scrape config into ScrapeConfig

33cc543

Translate cadvisor scrape config into ScrapeConfig

3401289

Translate garden scrape config into PodMonitor

1ee2ed7

Translate extensions scrape config into PodMonitor

ee96f89

Integrate seed prometheus deployment into Seed controller

0751914

Delete no longer needed code

7f38f6c

Address PR review feedback

34fa86e

Incorporate health of prometheus-seed MR in seed care controller

198b204

Address PR review feedback

1e29231

rfranzke force-pushed the gep19/seed-prometheus branch from 1bc2821 to 1e29231 Compare February 21, 2024 08:07

gardener-prow bot removed lgtm Indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Feb 21, 2024

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Feb 21, 2024

gardener-prow bot merged commit a0af468 into gardener:master Feb 21, 2024

rfranzke deleted the gep19/seed-prometheus branch February 21, 2024 09:32

This was referenced Feb 21, 2024

Do no longer delete prometheus-seed ClusterRoleBinding #9193

Merged

Update Plutono config for targeting seed Prometheus #9195

Merged

[GEP-19] Migrate aggregate Prometheus deployment and configuration #9200

Merged

[GEP-19] Migrate seed Prometheus deployment and configuration #9180

[GEP-19] Migrate seed Prometheus deployment and configuration #9180

Uh oh!

Conversation

rfranzke commented Feb 16, 2024

Uh oh!

ScheererJ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ScheererJ left a comment

Choose a reason for hiding this comment

Uh oh!

gardener-prow bot commented Feb 20, 2024

Uh oh!

oliver-goetz commented Feb 20, 2024

Uh oh!

oliver-goetz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

oliver-goetz Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

rfranzke Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

oliver-goetz Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

rfranzke Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

voelzmo Feb 21, 2024

Choose a reason for hiding this comment

Uh oh!

rfranzke commented Feb 20, 2024

Uh oh!

oliver-goetz commented Feb 20, 2024

Uh oh!

gardener-prow bot commented Feb 20, 2024

Uh oh!

gardener-prow bot commented Feb 20, 2024

Uh oh!

oliver-goetz commented Feb 21, 2024

Uh oh!

gardener-prow bot commented Feb 21, 2024

Uh oh!

Uh oh!