This repository was archived by the owner on Feb 18, 2025. It is now read-only.
ConsulTxnStore: batch KV updates by key-prefix to avoid ops limit #1311
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves issue: #1302
Description
This PR batches KVPair updates by key-prefix in to resolve errors for updates with more than 64 KVPairs. KVPairs are batched by key-prefix to ensure keys for a single "cluster" are updated in a single transaction atomically as only 5 x KVPair updates are necessary (< 64)
Details:
ConsulMaxKVsPerTransaction
config-param for increasing transaction size, default to5
5
causes 1 x transaction per cluster as there are 5 x KVs each5
(# of KVs/cluster) and64
(a Consul API limit).DistributePairs()
mysql/master/cluster1
,mysql/master/cluster1/hostname
,mysql/master/cluster1/ipv4
,mysql/master/cluster1/ipv6
andmysql/master/cluster1/port
is grouped into a single transaction at the defaultConsulMaxKVsPerTransaction=5
sync.WaitGroup
Transaction contains too many operations...
to mock Consul server in unit test.DistributePairs()
docs/kv.md
updateI'm not 100% sure if my changes to
go/config/config.go
fit with the theme. I also don't like that I made a constant for the number of KVPairs per "cluster". Any suggestions appreciated!Example debug-logging from
.DistributePairs()
:cc @shlomi-noach