Background

The current hotspot identification mainly uses byte and key dimensions, which are not very fit for load requests with high CPU overhead but low hard disk overhead, we plan to add QPS dimension to better describe the distribution of CPU resources.

After adding more dimension in the statistics, the hotspot scheduler can extract to a multi-objective optimized problem. For example, we have a simple workload about the IndexLoopUp scenario:

As you can see in the picture, the IO is very imbalanced but the QPS is balanced. In the hot read scenario, we priority to choose the QPS dimension because this scenario is CPU bound in most cases. consider there are many cases and the priority dimension may be changed in different cases, we need to support some relative dimension statistics and make the dimension's priority or weight can be turned.

Development Task

Reformat hot statistic and hot scheduler

• Support report hotspot in store heartbeat #10122 cluster,statistics: support recving peer stat by store heartbeat #3665
• Simplify pending influence scheduler: simplify pending influence #3658
• Replace move-hot-read-peer with move-hot-read-leader scheduler: replace move-hot-read-peer with move-hot-read-leader #3657
• Statistics: use []float64 to represent hot peer loads statistics: use []float64 to represent hot peer loads #3591
• Statistics: replace loads with []float64 in peer info statistics: replace loads with []float64 in peer info  #3729

Introduce QPS statistic

• Report QPS about each type of command in RegionHeartbeat and StoreHeartbeat
  Report QPS about each type of command in RegionHeartbeat and StoreHeartbeat tikv#8157
  Report QPS about each type of command in RegionHeartbeat and StoreHeartbeat #3695
  Report QPS about each type of command in RegionHeartbeat and StoreHeartbeat pingcap/kvproto#638

• Avoid merging regions with high QPS checker: avoid to merge region with high qps #3805

• scheduler: use qps priority with multi priorities in hot region scheduler scheduler: use qps priority with multi priorities in hot region scheduler  #3923

• scheduler: compatibility about query info in hot scheduler scheduler: compatibility about query info in hot scheduler #3947

• scheduler: use the store qps stats in the hot-write-leader-balance scheduler: use the store qps stats in the hot-write-leader-balance #3948

write qps

• Add more query kind in heartbeat
  add more query kind in heartbeat pingcap/kvproto#794
  add more query kind in heartbeat pingcap/kvproto#795
  statistics: add more write query kind #4011
  statistics: add more write query kind tikv#10786

Support multi priorities

• scheduler: support multi priorities api for hot-region-scheduler scheduler: support multi priorities api for hot-region-scheduler #3899
• Support multi priority scheduler: support priority for each dimension in hot scheduling #3920
• scheduler: set different priorities for write-leader and write-peer scheduler: set different priorities for write-leader and write-peer #3937
• scheduler: fix json tag for hot region scheduler config scheduler: fix json tag for hot region scheduler config #3942
• scheduler: adjust hot region config when pd-ctl return scheduler: adjust hot region config when pd-ctl return #3982
• scheduler: use compatible config after upgrading for hot-region scheduler: use compatible config after upgrading for hot-region #3981

Reduce different dim conflict

• scheduler: use pending amp in hot region scheduler scheduler: use pending amp in hot region scheduler #3926
• scheduler: unify leaderSolver and peerSolver in hot read scheduler scheduler: unify leaderSolver and peerSolver in hot read scheduler #3925

Misc

• scheduler: fix isHot check scheduler: fix isHot in checkHot #3722
• More Friendly hotspot info in API statistics: more friendly output #3731
• scheduler: fix sort hot peer in hot scheduler scheduler: fix sort hotpeer in hot scheduler #3741
• statistic: more stable test for query test statistic: more stable test for query test tikv#10386
• metrics: add qps metrics and remove unnecessary metrics metrics: add qps metrics and remove unnecessary metrics #3954
• scheduler: fix anticount and improving the Robustness of Hotspot Statistics scheduler: fix anticount and improving the Robustness of Hotspot Statistics #3928
• scheduler: fix inaccurate statistics and config scheduler: fix inaccurate statistics and config #3949
• scheduler: consider evict leader when calc expect scheduler: consider evict leader when calc expect #3967

Introduce the weight and priority

• TBD

Advance

• Report CPU usage to PD with the heartbeat tikv#10736