[GEP-28] Add support for NodeAgentAuthorizer feature to gardenadm init
#12169
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
42986da to
466795a
Compare
|
/assign |
rfranzke
requested changes
May 22, 2025
NodeAgentAuthorizer feature to gardenadm initNodeAgentAuthorizer feature to gardenadm init
|
/assign |
1f644c8 to
86b18c8
Compare
ScheererJ
reviewed
May 23, 2025
156ed78 to
ae5735e
Compare
rfranzke
added a commit
to oliver-goetz/gardener
that referenced
this pull request
Jun 3, 2025
See gardener#12169 (comment) for more details - requires temporary cluster-admin permissions since node-agent authorizer is not running yet - GNA now runs before `kubelet` is bootstrapped - this allows to remove the manual kubelet bootstrapping (GNA does it already) - at the end of the flow, the temporary cluster-admin permissions are removed, and it is ensured that GNA is still active/ready In addition, deploying the control plane deployments now also updates the OSC and the Secret reconciled by node-agent.
|
/hold (see #12169 (comment)) |
oliver-goetz
commented
Jun 5, 2025
Co-authored-by: Rafael Franzke <rafael.franzke@sap.com>
This is done by csr-approver now.
This step verifies, that gardener-node-agent is really running and not in a kind of crash loop. Additionally, this ensures that kube-apiserver has been rolled and the node-agent-authorizer webhook is accessible. Otherwise, gardener-node-agent would not be able to start.
Machine name must be set correctly to make this work. Additionally, the last-applied-osc.yaml file must not be written when GNA Reconciler applies the init OSC. Otherwise, it would delete the `machine-name` file right away because it is in the init OSC but not in the original one. While gardener-node-agent is bootstrapping there is no node in the `gardenadm join` scenario. Thus, it requires access to all OSC secrets because node-agent-authorizer does not know which worker pool the new GNA is using. If there would be one OSC per node, we could restrict the access to the OSC secrets which are not used on any node.
See gardener#12169 (comment) for more details - requires temporary cluster-admin permissions since node-agent authorizer is not running yet - GNA now runs before `kubelet` is bootstrapped - this allows to remove the manual kubelet bootstrapping (GNA does it already) - at the end of the flow, the temporary cluster-admin permissions are removed, and it is ensured that GNA is still active/ready In addition, deploying the control plane deployments now also updates the OSC and the Secret reconciled by node-agent.
07f3ea5 to
5be5846
Compare
|
/unhold |
|
/lgtm |
|
LGTM label has been added. Git tree hash: 77c0c67c095975c9b273140cb40ba85e97247253
|
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rfranzke The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
rfranzke
added a commit
to rfranzke/gardener
that referenced
this pull request
Jun 17, 2025
After gardener#12169, there is another restart of the control plane to activate the NodeAgentAuthorizer webhook (this changes configuration in `kube-apiserver`). Additionally, we have to ensure that `gardener-node-agent` is still running afterwards, and this can take some additional time. In some e2e runs, the current `5m` timeout is simply too short, see for example: - https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/12344/pull-gardener-e2e-kind-gardenadm/1934980463915438080 - https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/12335/pull-gardener-e2e-kind-gardenadm/1934970509573754880 - https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/12318/pull-gardener-e2e-kind-gardenadm/1934964955145048064 or generally https://prow.gardener.cloud/?repo=gardener%2Fgardener&job=pull-gardener-e2e-kind-gardenadm&state=failure, if you are fast enough
gardener-prow bot
pushed a commit
that referenced
this pull request
Jun 18, 2025
After #12169, there is another restart of the control plane to activate the NodeAgentAuthorizer webhook (this changes configuration in `kube-apiserver`). Additionally, we have to ensure that `gardener-node-agent` is still running afterwards, and this can take some additional time. In some e2e runs, the current `5m` timeout is simply too short, see for example: - https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/12344/pull-gardener-e2e-kind-gardenadm/1934980463915438080 - https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/12335/pull-gardener-e2e-kind-gardenadm/1934970509573754880 - https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/12318/pull-gardener-e2e-kind-gardenadm/1934964955145048064 or generally https://prow.gardener.cloud/?repo=gardener%2Fgardener&job=pull-gardener-e2e-kind-gardenadm&state=failure, if you are fast enough
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
How to categorize this PR?
/area ipcei
/kind enhancement
What this PR does / why we need it:
This PR adds support for the
NodeAgentAuthorizerfeature gate togardenadm init.In gardenadm high touch scenario there are no
Machines. Thus, this PR adds adds a new feature tocsr-approvercontroller andnode-agent-authorizerwebhook that they support a scenario with no machines too.Now,
csr-approverandnode-agent-authorizercan be enabled ingardener-resource-managerandgardener-node-agentis able to use client certificates for authorization.Additionally, this PR introduces a mutate function for the static pod translator (thanks to @rfranzke 😄). With this mutator function we can add a host alias for the service IP of GRM to kube-apiserver. This is required that kube-apiserver is able to reach the node-agent-authorizer webhook.
The initial client certificate for
gardener-node-agentneeds to be created in the gardenadm init flow before kubelet starts bootstrapping. We use the kubelet bootstrap token for this case which is invalided when kubelet has been bootstrapped successfully. In the shoot scenario gardener-node-agent deploys the kubelet. In gardenadm init flow the node-agent is deployed at the very end.Which issue(s) this PR fixes:
Part of #2906
Special notes for your reviewer:
Release note: