[GEP-28] Deploy `gardener-resource-manager` and extension controllers (in bootstrap mode) #11853

rfranzke · 2025-04-10T07:49:00Z

How to categorize this PR?

/area ipcei
/kind enhancement

What this PR does / why we need it:
This PR is the next increment for gardenadm init. It deploys gardener-resource-manager, the seed/shoot system components (which mostly are PriorityClasses, and the extension controllers in bootstrap mode).

Result:

root@machine-0:/# k get po,pc -A
NAMESPACE                     NAME                                                        READY   STATUS    RESTARTS   AGE
extension-networking-calico   pod/gardener-extension-networking-calico-6bb84d9879-zg4hm   1/1     Running   0          16m
extension-provider-local      pod/gardener-extension-provider-local-794c9976b7-tn8hx      1/1     Running   0          16m
kube-system                   pod/etcd-events-0-machine-0                                 1/1     Running   0          15m
kube-system                   pod/etcd-main-0-machine-0                                   1/1     Running   0          15m
kube-system                   pod/gardener-resource-manager-6f9cf4c994-2f8px              1/1     Running   0          17m
kube-system                   pod/kube-apiserver-machine-0                                1/1     Running   0          15m
kube-system                   pod/kube-controller-manager-machine-0                       1/1     Running   0          15m
kube-system                   pod/kube-scheduler-machine-0                                1/1     Running   0          15m

NAMESPACE   NAME                                                               VALUE        GLOBAL-DEFAULT   AGE   PREEMPTIONPOLICY
            priorityclass.scheduling.k8s.io/gardener-reserve-excess-capacity   -5           false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-shoot-system-600          999999600    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-shoot-system-700          999999700    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-shoot-system-800          999999800    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-shoot-system-900          999999900    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-100                999998100    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-200                999998200    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-300                999998300    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-400                999998400    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-500                999998500    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-600                999998600    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-700                999998700    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-800                999998800    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/gardener-system-900                999998900    false            16m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/system-cluster-critical            2000000000   false            17m   PreemptLowerPriority
            priorityclass.scheduling.k8s.io/system-node-critical               2000001000   false            17m   PreemptLowerPriority

Which issue(s) this PR fixes:
Part of #2906

Special notes for your reviewer:
/cc @ScheererJ

Release note:

NONE

ScheererJ · 2025-04-10T08:03:33Z

/assign

ScheererJ

Great progress. Thanks a lot.

The changes look very good already. I have only few comments/questions.

hack/generate-gardenadm-imagevector-overwrite.sh

pkg/component/gardener/resourcemanager/resource_manager.go

pkg/gardenadm/botanist/botanist.go

pkg/gardenadm/resources.go

pkg/gardenadm/botanist/extensions.go

pkg/utils/gardener/shoot.go

ScheererJ

/lgtm
/approve

gardener-prow · 2025-04-11T14:20:20Z

LGTM label has been added.

Git tree hash: 556ce0f82bde3d23822340b7c5a38fa3a24d7619

ScheererJ · 2025-04-14T09:51:40Z

/test pull-gardener-e2e-kind-gardenadm

ScheererJ

/lgtm
/approve

gardener-prow · 2025-04-15T10:28:01Z

LGTM label has been added.

Git tree hash: 8de0ae3f0a83fd817c52e44f2f237d33a7f1bfc1

gardener-prow · 2025-04-15T10:28:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ScheererJ

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ScheererJ]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ScheererJ · 2025-04-16T07:05:06Z

/lgtm

This way, we can easily execute `kubectl` commands in the machine pods without explicitly pointing to the admin kubeconfig file.

This results in a more convenient use (the only use-case for this static token kubeconfig are the autonomous shoot clusters, and here we usually operate in the `kube-system` namespace, i.e., with this change, we can simply do `kubectl get pods` and see all pods in `kube-system`).

This includes - priority classes - read-only clusterrole + clusterrolebindings (for viewers) - some network policies - KCM service accounts - shoot-info config map

- only consider injecting garden kubeconfig when "bootstrap" flag is disabled - prevent nil pointer exceptions by checking explicitly if ingress domain is set - change rolling update strategy for extension deployments to `Recreate` (to avoid host port conflicts) - set `KUBERNETES_SERVICE_HOST` env var to `localhost` (extensions get deployed before kube-proxy, so the default kubernetes cluster IP does not work yet)

- disable coredns deployment (we don't need it in `gardenadm` scenarios) - do not depend on garden `cluster.Cluster` object (this is only needed for shoot control plane migration at the moment) - inject correct image ref built via skaffold into `ControllerDeployment`

- the list of ControllerRegistrations/ControllerDeployments can be quite long (later, this can be downloaded from an existing Gardener system via `gardenadm discover`, i.e., there can be a lot of extension) - we should only deploy those we really need

Otherwise, the `kubelet` might fail for a while to create the mirror pods for the static pods because a webhook might not be available yet. Since the webhooks are anyways not needed for the mirror pods, we don't want them to be called in the first place.

ScheererJ · 2025-04-16T10:17:57Z

/lgtm

gardener-prow · 2025-04-16T10:18:02Z

LGTM label has been added.

Git tree hash: 46c5f8973a199f1d2f786667d3b99e964502f509

- With gardener@f9f6a46 (part of gardener#11853) we only skipped static pods during the bootstrap phase. However, we actually never want GRM webhooks to react on static pod creations, even if we are beyond the bootstrap phase. Otherwise, the kubelet might fail to create mirror pods for control plane components if the GRM webhook is unavailable. This can lead to undesired behaviour.

…d` (#12038) * Deploy `etcd-druid` CRDs * Deploy `etcd-druid` * Static pod translator: Remove service account names This is not allowed for mirror pods * Static pod translator: Translate STS volume claim templates We just mount a path from the host instead. * Prepare bootstrap etcd - change names from `etcd-<role>-0` to `etcd-bootstrap-<role>` to make it more clear when `kubectl get pod` - use the same data directory that etcd managed by etcd-druid would use later (it shall just take over the existing data dir) * GRM pod-creation webhooks always skip static pods - With f9f6a46 (part of #11853) we only skipped static pods during the bootstrap phase. However, we actually never want GRM webhooks to react on static pod creations, even if we are beyond the bootstrap phase. Otherwise, the kubelet might fail to create mirror pods for control plane components if the GRM webhook is unavailable. This can lead to undesired behaviour. * Bootstrap etcd must use same version as etcd-druid - Otherwise, we might get issues when etcd-druid takes over etcd management later - Review this commit with `git diff -w` (ignoring whitespaces) - `yq` seems to enforce indentation, so let's live with it Output: $ go generate ./imagevector/imagevector.go containers.go Cloning etcd-druid... Found etcd-wrapper tag: v0.4.4 Cloning etcd-wrapper at tag v0.4.4... Found etcd version string: v0.0.0-20240911181550-c123b3ea3db3 Extracted commit hash: c123b3ea3db3 Cloning etcd repo to resolve tag... Resolved etcd tag: v3.4.34 Updating containers.yaml with etcd v3.4.34 tag charts.go

gardener-prow bot requested a review from ScheererJ April 10, 2025 07:49

rfranzke mentioned this pull request Apr 10, 2025

☂️ [GEP-28] Autonomous Shoot Clusters #2906

Open

gardener-prow bot assigned ScheererJ Apr 10, 2025

rfranzke force-pushed the gep28/grm branch 2 times, most recently from 3da5071 to 252920f Compare April 10, 2025 10:47

ScheererJ reviewed Apr 10, 2025

View reviewed changes

rfranzke force-pushed the gep28/grm branch from 252920f to 2720ea2 Compare April 11, 2025 06:45

rfranzke requested a review from ScheererJ April 11, 2025 06:45

ScheererJ approved these changes Apr 11, 2025

View reviewed changes

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Apr 11, 2025

gardener-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 11, 2025

rfranzke force-pushed the gep28/grm branch from 2720ea2 to 066ef4b Compare April 15, 2025 10:24

gardener-prow bot removed the lgtm Indicates that a PR is ready to be merged. label Apr 15, 2025

gardener-prow bot requested a review from ScheererJ April 15, 2025 10:24

ScheererJ approved these changes Apr 15, 2025

View reviewed changes

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Apr 15, 2025

rfranzke force-pushed the gep28/grm branch from 066ef4b to fac1b0d Compare April 16, 2025 07:04

gardener-prow bot removed the lgtm Indicates that a PR is ready to be merged. label Apr 16, 2025

gardener-prow bot requested a review from ScheererJ April 16, 2025 07:04

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Apr 16, 2025

gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 16, 2025

rfranzke added 14 commits April 16, 2025 12:15

Nit: Set KUBECONFIG env var in machine pods

234a6f9

This way, we can easily execute `kubectl` commands in the machine pods without explicitly pointing to the admin kubeconfig file.

Nit: Also set ShootClientSet in botanist

32aedd7

Nit: NewHelmRegistry no longer returns error

6d72749

Build gardener-resource-manager in skaffold-gardenadm.yaml

1c663b9

Support resource-manager reconciliation on all resource classes

6d8a227

Deploy seed and shoot system components

cc9f48e

This includes - priority classes - read-only clusterrole + clusterrolebindings (for viewers) - some network policies - KCM service accounts - shoot-info config map

Read and parse Controller{Registration,Deployment} manifests

17b4c85

Deploy extension controllers to host network

3cdc71b

Address PR review feedback

fcd6613

rfranzke force-pushed the gep28/grm branch from fac1b0d to f9f6a46 Compare April 16, 2025 10:17

gardener-prow bot removed lgtm Indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Apr 16, 2025

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Apr 16, 2025

gardener-prow bot merged commit 2a8498c into gardener:master Apr 16, 2025
19 checks passed

rfranzke deleted the gep28/grm branch April 16, 2025 14:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[GEP-28] Deploy `gardener-resource-manager` and extension controllers (in bootstrap mode) #11853

[GEP-28] Deploy `gardener-resource-manager` and extension controllers (in bootstrap mode) #11853

Uh oh!

rfranzke commented Apr 10, 2025

Uh oh!

ScheererJ commented Apr 10, 2025

Uh oh!

ScheererJ left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ScheererJ left a comment

Uh oh!

gardener-prow bot commented Apr 11, 2025

Uh oh!

ScheererJ commented Apr 14, 2025

Uh oh!

ScheererJ left a comment

Uh oh!

gardener-prow bot commented Apr 15, 2025

Uh oh!

gardener-prow bot commented Apr 15, 2025

Uh oh!

ScheererJ commented Apr 16, 2025

Uh oh!

ScheererJ commented Apr 16, 2025

Uh oh!

gardener-prow bot commented Apr 16, 2025

Uh oh!

Uh oh!

Uh oh!

[GEP-28] Deploy gardener-resource-manager and extension controllers (in bootstrap mode) #11853

[GEP-28] Deploy gardener-resource-manager and extension controllers (in bootstrap mode) #11853

Uh oh!

Conversation

rfranzke commented Apr 10, 2025

Uh oh!

ScheererJ commented Apr 10, 2025

Uh oh!

ScheererJ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ScheererJ left a comment

Choose a reason for hiding this comment

Uh oh!

gardener-prow bot commented Apr 11, 2025

Uh oh!

ScheererJ commented Apr 14, 2025

Uh oh!

ScheererJ left a comment

Choose a reason for hiding this comment

Uh oh!

gardener-prow bot commented Apr 15, 2025

Uh oh!

gardener-prow bot commented Apr 15, 2025

Uh oh!

ScheererJ commented Apr 16, 2025

Uh oh!

ScheererJ commented Apr 16, 2025

Uh oh!

gardener-prow bot commented Apr 16, 2025

Uh oh!

Uh oh!

Uh oh!

[GEP-28] Deploy `gardener-resource-manager` and extension controllers (in bootstrap mode) #11853

[GEP-28] Deploy `gardener-resource-manager` and extension controllers (in bootstrap mode) #11853