Skip to content

operator: skip node taint retry when node not found #39517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 15, 2025
Merged

operator: skip node taint retry when node not found #39517

merged 1 commit into from
May 15, 2025

Conversation

jshr-w
Copy link
Contributor

@jshr-w jshr-w commented May 14, 2025 •
edited
Loading

The Cilium Operator logs are filled with attempts to retry updating taints/conditions on nodes even if we can't get the node from the local store due to node deletion. Skip retry in this case.

Fixes: #38059

operator: skip retry of node taint update when node not found

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 14, 2025
@jshr-w
Copy link
Contributor Author

jshr-w commented May 14, 2025

/test

@jshr-w
Copy link
Contributor Author

jshr-w commented May 14, 2025 •
edited
Loading

Tested this at scale, and it seems to clear out the log issue. Chose to skip retry since this seems to align with the original behavior of this code:

cilium/operator/watchers/node_taint.go

Lines 91 to 94 in 0e4bc5e

node, err := nodeGetter.GetK8sSlimNode(nodeName)
if node == nil || err != nil {
return false
}

cilium/operator/watchers/node_taint.go

Lines 74 to 78 in 0e4bc5e

success := checkAndMarkNode(c, nodeGetter, key.(string), mno, logger)
if !success {
workQueue.Forget(key)
return true
}

@jshr-w
operator: skip node taint retry when node not found
1a2d7f4 
The Cilium Operator logs are filled with attempts to retry updating
taints/conditions on nodes even if we can't get the node from the local
store due to node deletion. Skip retry in this case.

Signed-off-by: jshr-w <shjayaraman@microsoft.com>
@jshr-w jshr-w force-pushed the jshr/nodenotfound branch from 395fd4c to 1a2d7f4 Compare May 14, 2025 19:28
@jshr-w jshr-w marked this pull request as ready for review May 14, 2025 20:23
@jshr-w jshr-w requested a review from a team as a code owner May 14, 2025 20:23
@jshr-w jshr-w requested a review from pippolo84 May 14, 2025 20:23
@jshr-w
Copy link
Contributor Author

jshr-w commented May 14, 2025

/test

@pippolo84 pippolo84 added area/operator Impacts the cilium-operator component release-note/misc This PR makes changes that have no direct user impact. labels May 15, 2025
@maintainer-s-little-helper maintainer-s-little-helper bot removed dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. labels May 15, 2025
@pippolo84 pippolo84 added kind/bug This is a bug in the Cilium logic. release-note/bug This PR fixes an issue in a previous release of Cilium. needs-backport/1.17 This PR / issue needs backporting to the v1.17 branch and removed release-note/misc This PR makes changes that have no direct user impact. kind/bug This is a bug in the Cilium logic. labels May 15, 2025
pippolo84
pippolo84 approved these changes May 15, 2025
View reviewed changes
Copy link
Member

@pippolo84 pippolo84 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@pippolo84 pippolo84 mentioned this pull request May 15, 2025
Closed
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label May 15, 2025
@julianwiedmann julianwiedmann added this pull request to the merge queue May 15, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 15, 2025
@julianwiedmann julianwiedmann added this pull request to the merge queue May 15, 2025
Merged via the queue into cilium:main with commit 9938f52 May 15, 2025
70 checks passed
@nbusseneau nbusseneau mentioned this pull request May 15, 2025
Merged
9 tasks
@nbusseneau nbusseneau added backport-pending/1.17 The backport for Cilium 1.17.x for this PR is in progress. and removed needs-backport/1.17 This PR / issue needs backporting to the v1.17 branch labels May 15, 2025
@github-actions github-actions bot removed the backport-pending/1.17 The backport for Cilium 1.17.x for this PR is in progress. label May 19, 2025
@github-actions github-actions bot added the backport-done/1.17 The backport for Cilium 1.17.x for this PR is done. label May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pippolo84 pippolo84 pippolo84 approved these changes

Assignees
No one assigned
Labels
area/operator Impacts the cilium-operator component backport-done/1.17 The backport for Cilium 1.17.x for this PR is done. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/bug This PR fixes an issue in a previous release of Cilium.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Multiple consecutive retries of updating taints and conditions for a node failed
4 participants
@jshr-w @pippolo84 @nbusseneau @julianwiedmann