How to categorize this issue?

/area dev-productivity monitoring
/kind enhancement

What would you like to be added:
The monitoring stack should be migrated from the current custom-built Helm charts to the prometheus-operator as proposed in GEP-19.

Why is this needed:
GEP-19 has been accepted and merged a long while ago, hence we should strive for completing its implementation. Also, the garden cluster (managed via gardener-operator, ref #7016) does not have a monitoring stack yet. Also, this increases the development productivity by cleaning up technical debt and improving the code.

Tasks:

• Deployment of prometheus-operator
  • Golang component package: [GEP-19] Introduce prometheus-operator in garden and seed clusters #9067
  • Deployment to garden cluster via gardener-operator: [GEP-19] Introduce prometheus-operator in garden and seed clusters #9067
  • Deployment to seed clusters via gardenlet: [GEP-19] Introduce prometheus-operator in garden and seed clusters #9067
• Prometheis/Alertmanager responsible for seed cluster
  • [GEP-19] Migrate cache Prometheus deployment and configuration #9128
    • [GEP-19] Fix cache Prometheus scrape config for kube-state-metrics #9179
    • Fix VPA for Prometheis managed by prometheus-operator #9188
    • Seed care controller incorporates health of prometheus-cache and alertmanager-seed ManagedResources #9189
  • [GEP-19] Migrate seed Prometheus deployment and configuration #9180
    • Do no longer delete prometheus-seed ClusterRoleBinding #9193
    • Update Plutono config for targeting seed Prometheus #9195
  • [GEP-19] Migrate aggregate Prometheus deployment and configuration #9200
    • [GEP-19] Delete networkingv1.Ingress instead of networkingv1beta1.Ingress #9299
  • [GEP-19] Migrate seed Alertmanager deployment and configuration #9159
    • Seed care controller incorporates health of prometheus-cache and alertmanager-seed ManagedResources #9189
    • Copy alerting SMTP secret from garden to seed cluster #9190
    • [GEP-19] Set replicas=1 for seed alertmanager #9298
    • [GEP-19] Fix the match expression in the alertmanager configuration #9387
• Prometheis/Alertmanager responsible for garden cluster
  • [GEP-19] Integrate Alertmanager deployment into Garden controller #9301
  • [GEP-19] Integrate Prometheus and blackbox-exporter deployments into Garden controller #9543
    • [GEP-19] Allow public network access for Garden Prometheus #9587
  • [GEP-19] Integrate long-term Prometheus deployment into Garden controller #9606
• Prometheus/Alertmanager responsible for shoot clusters
  • [GEP-19] Migrate shoot Alertmanager deployment and configuration #9257
    • [GEP-19] [shoot-care] Check for alertmanager-shoot when Gardener >= 1.90 #9335
    • [GEP-19] Add namespace to value of PVC migration label in PersistentVolumes #9338
    • [GEP-19] Add dedicated namespace label for PVCs in migration #9344
    • [GEP-19] Ensure legacy resource cleanup for shoot Alertmanager is executed #9416
  • [GEP-19] Migrate shoot Prometheus deployment #9695
  • Adapt shoot Prometheus configuration (scrape configs, rules, ...) for components deployed by gardenlet
    • [GEP-19] Adapt monitoring configuration for shoot cluster system components #9737
    • [GEP-19] Adapt monitoring configuration for shoot control plane components #9848
• Adapt how extensions provide their observability configuration
  • provider-alicloud: [GEP-19] Adapt monitoring configuration gardener-extension-provider-alicloud#720
  • provider-aws: [GEP-19] Adapt monitoring configuration gardener-extension-provider-aws#946
  • provider-azure: [GEP-19] Adapt monitoring configuration gardener-extension-provider-azure#853
  • provider-gcp: [GEP-19] Adapt monitoring configuration gardener-extension-provider-gcp#754
  • provider-openstack: [GEP-19] Adapt monitoring configuration gardener-extension-provider-openstack#766
  • provider-equinix-metal: [GEP-19] Adapt monitoring configuration gardener-extension-provider-equinix-metal#307
  • networking-calico: [GEP-19] Adapt monitoring configuration gardener-extension-networking-calico#394
  • networking-cilium: [GEP-19] Adapt monitoring configuration gardener-extension-networking-cilium#307
  • shoot-cert-service: [GEP-19] Adapt monitoring configuration gardener-extension-shoot-cert-service#257
  • shoot-oidc-service: [GEP-19] Adapt monitoring configuration gardener-extension-shoot-oidc-service#193
  • shoot-lakom-service: [GEP-19] Adapt monitoring configuration gardener-extension-shoot-lakom-service#87
  • shoot-networking-problemdetector: [GEP-19] Adapt monitoring configuration gardener-extension-shoot-networking-problemdetector#142
  • shoot-rsyslog-relp: [GEP-19] Adapt monitoring configuration gardener-extension-shoot-rsyslog-relp#99
  • registry-cache: [GEP-19] Switch to the new contract of providing monitoring configuration gardener-extension-registry-cache#187
• Miscellaneous
  • [GEP-19] Extend health checks of gardener-resource-manager for new Prometheus and Alertmanager resources #9163
  • Consider deployment of admission webhook server (abandoned for now due to other, more important topics)
  • [GEP-19] Add sidecar to Plutono for fetching dashboard ConfigMaps dynamically #9624

General notes for the migration (taken from #6319):

• Add temporary migration code for the Persistent volume. This ensures that no data is lost.
  1. Find the "old" pvc and its pv and set persistentVolumeReclaimPolicy=Retain.
  2. Delete the "old" pvc.
  3. Create a Prometheus Object with a volumeClaimTemplate that references the pv with volumeName=<existing-pv>
  4. Migrate the data using an init container
  5. Remove the migration code after 1-2 releases
• Add all existing prometheus configuration to an additionalScrapeConfig. This will allow us to switch to the prometheus-operator without creating PodMonitors and ServiceMonitors for each component and instead do that migration step by step.
• Add all extension prometheus configuration to the same additionalScrapeConfig. This will allow extensions time to migrate as well.
• Existing rules should be replaced with PrometheusRules.
• Once all of these steps are completed, most of the configuration in the additionalScrapeConfig can be migrated to PodMonitors and ServiceMonitors.