After real-world testing of the new transaction-based Consul KV store I ran into errors when more than 64 KVs were read or updated:

consulTxnStore.doWriteTxn(): Failed request: Transaction contains too many operations (525 > 64)

And indeed the Consul Transaction docs mention this limit, something I missed in #1276 because it's only mentioned casually in the notes for an example payload, not the initial API description:

The body of the request should be a list of operations to perform inside the atomic transaction. Up to 64 operations may be present in a single transaction.

This issue is to track the need for these updates to be batched in groups of less than 64 operations. The 4 x KVs (hostname,ipv4, ipv6 and port) for a single cluster will need to remain isolated to a single transaction "group" to remain atomic

My plan to resolve this:

1. Group the KVPair updates by key-prefix, ensuring atomic updates for a single cluster (same Key-prefix)
2. Add a ConsulMaxKVsPerTransaction config field with an acceptable range from 4 to 64, with a default: 4 (1 x transaction per cluster)
   • Increasing this value will reduce calls to Consul but increases likelihood of a rollback
3. Perform reads/updates using grouped operations

A PR to fix this is coming soon!