Fix shoot reconciliation after credentials rotation if manual in-place pending workers are updated #12290

shafeeqes · 2025-06-11T09:29:50Z

How to categorize this PR?

/area quality
/kind bug

What this PR does / why we need it:
In the Shoot reconciliation flow, we don't set the credentialsRotation status to Prepared if there are manual in-place workers pending update. When they are updated, shoot status reconciler will annotate the shoot with gardener.cloud/operation=reconcile,

gardener/pkg/gardenlet/controller/shoot/status/reconciler.go

Lines 163 to 175 in 4c7ae4a

    
           if shootNeedsReconcile || (noInPlacePendingWorkers && kubernetesutils.HasMetaDataAnnotation(shoot, v1beta1constants.GardenerOperation, v1beta1constants.ShootOperationForceInPlaceUpdate)) { 
        
           	patch := client.MergeFromWithOptions(shoot.DeepCopy(), client.MergeFromWithOptimisticLock{}) 
        
           	if shootNeedsReconcile { 
        
           		log.Info("Triggering a Shoot reconciliation after credential rotations for manual in-place workers are prepared") 
        
           		metav1.SetMetaDataAnnotation(&shoot.ObjectMeta, v1beta1constants.GardenerOperation, v1beta1constants.GardenerOperationReconcile) 
        
           	} else { 
        
           		delete(shoot.Annotations, v1beta1constants.GardenerOperation) 
        
           	} 
        
           	if err := r.GardenClient.Patch(ctx, shoot, patch); err != nil { 
        
           		return reconcile.Result{}, fmt.Errorf("failed to patch Shoot: %w", err) 
        
           	} 
        
           }

so that the reconcile flow is run again and the status is set to Prepared. However if the shoot status is updated with zero manual in place update pending workers,

gardener/pkg/gardenlet/controller/shoot/status/reconciler.go

Lines 153 to 156 in 4c7ae4a

    
           log.Info("Updating Shoot status with manual in-place pending workers", "manualInPlacePendingWorkers", sets.List(manualInPlacePendingWorkers)) 
        
           if err := r.GardenClient.Status().Patch(ctx, shoot, patch); err != nil { 
        
           	return reconcile.Result{}, fmt.Errorf("failed to patch Shoot status: %w", err) 
        
           }

and the above patching of shoot with operation annotation fails, the next time the reconciliation is retried, we exit early:

gardener/pkg/gardenlet/controller/shoot/status/reconciler.go

Lines 79 to 82 in 4c7ae4a

    
           // If there are no manual in-place update workers in the shoot status, then nothing to do 
        
           if shoot.Status.InPlaceUpdates == nil || shoot.Status.InPlaceUpdates.PendingWorkerUpdates == nil || len(shoot.Status.InPlaceUpdates.PendingWorkerUpdates.ManualInPlaceUpdate) == 0 { 
        
           	return reconcile.Result{}, nil 
        
           }

This is not ideal and can cause failures like the one below.
This PR fixes this behaviour.

Thanks @rfranzke for reporting.

Which issue(s) this PR fixes:
Fixes failures like: https://prow.gardener.cloud/view/gs/gardener-prow/pr-logs/pull/gardener_gardener/12213/pull-gardener-e2e-kind-ha-multi-zone/1932519479678341120

Special notes for your reviewer:

Release note:

An issue causing the Shoot credentials rotation status not to correctly get updated, after all the manual in-place pending workers are updated, is now fixed.

shafeeqes · 2025-06-11T09:52:42Z

/cherry-pick release-v1.121

gardener-ci-robot · 2025-06-11T09:52:44Z

@shafeeqes: once the present PR merges, I will cherry-pick it on top of release-v1.121 in a new PR and assign it to you.

In response to this:

/cherry-pick release-v1.121

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

shafeeqes · 2025-06-11T09:53:14Z

/cherry-pick release-v1.120

gardener-ci-robot · 2025-06-11T09:53:17Z

@shafeeqes: once the present PR merges, I will cherry-pick it on top of release-v1.120 in a new PR and assign it to you.

In response to this:

/cherry-pick release-v1.120

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

shafeeqes · 2025-06-11T09:53:49Z

/cherry-pick release-v1.119

gardener-ci-robot · 2025-06-11T09:53:52Z

@shafeeqes: once the present PR merges, I will cherry-pick it on top of release-v1.119 in a new PR and assign it to you.

In response to this:

/cherry-pick release-v1.119

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

shafeeqes · 2025-06-11T09:53:54Z

/cherry-pick release-v1.118

gardener-ci-robot · 2025-06-11T09:53:57Z

@shafeeqes: once the present PR merges, I will cherry-pick it on top of release-v1.118 in a new PR and assign it to you.

In response to this:

/cherry-pick release-v1.118

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pkg/gardenlet/controller/shoot/status/reconciler.go

rfranzke

/lgtm
/approve
/retest

gardener-prow · 2025-06-11T13:31:51Z

LGTM label has been added.

Git tree hash: 477952c0b58543771a3051f95c8296f91559a121

gardener-prow · 2025-06-11T13:31:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ary1992 · 2025-06-12T01:52:35Z

/retest

shafeeqes · 2025-06-12T04:41:22Z

/test pull-gardener-e2e-kind-ha-multi-zone

gardener-ci-robot · 2025-06-12T06:17:38Z

@shafeeqes: new pull request created: #12303

In response to this:

/cherry-pick release-v1.120

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

gardener-ci-robot · 2025-06-12T06:18:18Z

@shafeeqes: new pull request created: #12304

In response to this:

/cherry-pick release-v1.119

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

gardener-ci-robot · 2025-06-12T06:18:59Z

@shafeeqes: new pull request created: #12305

In response to this:

/cherry-pick release-v1.118

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

gardener-ci-robot · 2025-06-12T06:19:38Z

@shafeeqes: new pull request created: #12306

In response to this:

/cherry-pick release-v1.121

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Fix shoot reconciliation after credentials rotation

316d17d

gardener-prow bot added area/quality Output qualification (tests, checks, scans, automation in general, etc.) related kind/bug Bug cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 11, 2025

gardener-prow bot requested review from oliver-goetz and timebertt June 11, 2025 09:30

shafeeqes mentioned this pull request Jun 11, 2025

☂️ [GEP-31] Support for In-Place Node Updates #10219

Open

40 tasks

rfranzke reviewed Jun 11, 2025

View reviewed changes

pkg/gardenlet/controller/shoot/status/reconciler.go Outdated Show resolved Hide resolved

Address PR review feedback

3a816ea

shafeeqes requested a review from rfranzke June 11, 2025 13:06

rfranzke reviewed Jun 11, 2025

View reviewed changes

gardener-prow bot assigned rfranzke Jun 11, 2025

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jun 11, 2025

gardener-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 11, 2025

gardener-prow bot merged commit 4a7b32d into gardener:master Jun 12, 2025
19 checks passed

gardener-ci-robot mentioned this pull request Jun 12, 2025

[release-v1.120] Fix shoot reconciliation after credentials rotation if manual in-place pending workers are updated #12303

Merged

gardener-ci-robot mentioned this pull request Jun 12, 2025

[release-v1.119] Fix shoot reconciliation after credentials rotation if manual in-place pending workers are updated #12304

Merged

gardener-ci-robot mentioned this pull request Jun 12, 2025

[release-v1.118] Fix shoot reconciliation after credentials rotation if manual in-place pending workers are updated #12305

Merged

gardener-ci-robot mentioned this pull request Jun 12, 2025

[release-v1.121] Fix shoot reconciliation after credentials rotation if manual in-place pending workers are updated #12306

Merged

shafeeqes deleted the fix/shoot-status branch June 12, 2025 06:40

	if shootNeedsReconcile \|\| (noInPlacePendingWorkers && kubernetesutils.HasMetaDataAnnotation(shoot, v1beta1constants.GardenerOperation, v1beta1constants.ShootOperationForceInPlaceUpdate)) {
	patch := client.MergeFromWithOptions(shoot.DeepCopy(), client.MergeFromWithOptimisticLock{})
	if shootNeedsReconcile {
	log.Info("Triggering a Shoot reconciliation after credential rotations for manual in-place workers are prepared")
	metav1.SetMetaDataAnnotation(&shoot.ObjectMeta, v1beta1constants.GardenerOperation, v1beta1constants.GardenerOperationReconcile)
	} else {
	delete(shoot.Annotations, v1beta1constants.GardenerOperation)
	}

	if err := r.GardenClient.Patch(ctx, shoot, patch); err != nil {
	return reconcile.Result{}, fmt.Errorf("failed to patch Shoot: %w", err)
	}
	}

	log.Info("Updating Shoot status with manual in-place pending workers", "manualInPlacePendingWorkers", sets.List(manualInPlacePendingWorkers))
	if err := r.GardenClient.Status().Patch(ctx, shoot, patch); err != nil {
	return reconcile.Result{}, fmt.Errorf("failed to patch Shoot status: %w", err)
	}

	// If there are no manual in-place update workers in the shoot status, then nothing to do
	if shoot.Status.InPlaceUpdates == nil \|\| shoot.Status.InPlaceUpdates.PendingWorkerUpdates == nil \|\| len(shoot.Status.InPlaceUpdates.PendingWorkerUpdates.ManualInPlaceUpdate) == 0 {
	return reconcile.Result{}, nil
	}

Fix shoot reconciliation after credentials rotation if manual in-place pending workers are updated #12290

Fix shoot reconciliation after credentials rotation if manual in-place pending workers are updated #12290

Uh oh!

Conversation

shafeeqes commented Jun 11, 2025

Uh oh!

shafeeqes commented Jun 11, 2025

Uh oh!

gardener-ci-robot commented Jun 11, 2025

Uh oh!

shafeeqes commented Jun 11, 2025

Uh oh!

gardener-ci-robot commented Jun 11, 2025

Uh oh!

shafeeqes commented Jun 11, 2025

Uh oh!

gardener-ci-robot commented Jun 11, 2025

Uh oh!

shafeeqes commented Jun 11, 2025

Uh oh!

gardener-ci-robot commented Jun 11, 2025

Uh oh!

Uh oh!

rfranzke left a comment

Choose a reason for hiding this comment

Uh oh!

gardener-prow bot commented Jun 11, 2025

Uh oh!

gardener-prow bot commented Jun 11, 2025

Uh oh!

ary1992 commented Jun 12, 2025

Uh oh!

shafeeqes commented Jun 12, 2025

Uh oh!

Uh oh!

gardener-ci-robot commented Jun 12, 2025

Uh oh!

gardener-ci-robot commented Jun 12, 2025

Uh oh!

gardener-ci-robot commented Jun 12, 2025

Uh oh!

gardener-ci-robot commented Jun 12, 2025

Uh oh!

Uh oh!