Skip to content

Support for resource patching with retries #877

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 12, 2025

Conversation

videlov
Copy link
Contributor

@videlov videlov commented Mar 27, 2025

For our scenario we need to scale Reloader to multiple Pods and the current HA implementation causes problems while we need zero downtime and leader election can not provide that with the current implementation.

Solution for us to increase resilience is to include patch on resources which will reduce conflicts possibility when multiple clients are trying. Also it solves the case for multiple restarts due to patching based on CM or Secret hash value, which will be same for all clients patch requests and one restart should be triggered.

For non-patch possible workloads, we introduce retries on conflicts, which will help to avoid missed updates.

PR introduces:

  • Introduce retry.RetryOnConflict in PerformAction for the upgrade handler
  • Add support for patching resources in rolling upgrade callbacks
  • Unit tests and manual testing to confirm our use case is solved

We would appreciate your comments and review. Thanks in advance.

@Nuckal777
Copy link

Ref #766.

@Felix-Stakater
Copy link
Contributor

Hi! We looked through this and are satisfied so far, nice job!
Would you mind rebasing your branch on our latest master? (requirement for the pipelines to run)

You can ping me here or in slack and I'll make sure to run them as soon as i see it. :)

* Initial

* State

* State

* Revert unintentional change

* Update

* Update

* Fix tests

* Support patching for both reload strategies

* rolling_upgrade tests

* Update

* More tests

* Remove unnecessary stuff
@videlov
Copy link
Contributor Author

videlov commented May 7, 2025

@Felix-Stakater thank you for looking into it. I did rebase to current master, and fixed lint issues.

@Felix-Stakater Felix-Stakater merged commit 5499911 into stakater:master May 12, 2025
5 checks passed
@Felix-Stakater
Copy link
Contributor

Thank you! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants