Problems with rebalance-threshold #26012
Description
Given the recommendation of using 10x shards than number of nodes the default rebalance-threshold=10 is too hight.
It can result in one node hosting ~2 times the number of shards of other nodes. Example: 1000 shards on 100 nodes means 10 shards per node. One node may have 19 shards and others 10 without a rebalance occurring.
There is also a bug possibly causing continuous rebalance if rebalance-threshold=1.
Here is a reproducer test that can be added to LeastShardAllocationStrategy:
"not rebalance one" in {
val allocations = Map(regionA → Vector("shard1"), regionB → Vector("shard2"), regionC → Vector.empty)
val strategy = new LeastShardAllocationStrategy(rebalanceThreshold = 1, maxSimultaneousRebalance = 2)
strategy.rebalance(allocations, Set.empty).futureValue should ===(Set.empty[String])
val allocations2 = Map(regionA → Vector.empty, regionB → Vector("shard2"), regionC → Vector("shard1"))
strategy.rebalance(allocations2, Set.empty).futureValue should ===(Set.empty[String])
}