statedb: Reconciler utility #30303
Merged
statedb: Reconciler utility #30303
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tommyp1ckles
approved these changes
Jan 17, 2024
|
/test |
28 tasks
dylandreimerink
approved these changes
Jan 22, 2024
fbde633 to
8b2e974
Compare
|
/test |
Tests that check that a module status is properly updated incur a 500ms wait. To remedy this, add a Realize() method to Scope that does immediate synchronous realization of the status. Signed-off-by: Jussi Maki <jussi@isovalent.com>
8b2e974 to
c0297fc
Compare
|
/test |
This implements a generic utility for reconciling a StateDB table and a target defined as a set of idempotent operations. It implements incremental reconciliation with per-object retries and a periodic full reconciliation (forced per-object update and pruning). A single shared reconciler implementation helps building a more resilient architecture as it centralizes the decisions on how to deal with failures and how to report them. Signed-off-by: Jussi Maki <jussi@isovalent.com>
Example application for showing how to use the reconciler
to reconcile set of files on disk ("memos") created through
an HTTP API.
Signed-off-by: Jussi Maki <jussi@isovalent.com>
A simple benchmark program for the generic reconciler.
Example run:
benchmark % go build .
benchmark % ./benchmark -objects=100000 -batchsize=1000
Inserting batch 100/100 ...
Waiting for reconciliation to finish ...
100000 objects reconciled in 0.35 seconds (batch size 1000)
Throughput 283977.47 objects per second
Signed-off-by: Jussi Maki <jussi@isovalent.com>
c0297fc to
4ae2c7f
Compare
|
/test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the statedb reconciler, a utility for reconciling a StateDB table with a target system defined by the operations Update/Delete/Prune.
The reconciler performs incremental reconciliation as objects change as well periodic full reconciliation to catch cases where the target may have diverged. Metrics are gathered for errors and the durations of the operations. Module health is reported and marked degraded if operations fail.