Skip to content

scheduler: balance hot stores formed a lot of cold peers  #5602

@lhy1024

Description

@lhy1024

Feature Request

Describe your feature request related problem

In some tests, we meet the problem which needs to balance hot stores formed a lot of cold peers.

There are three stores with higher qps than other stores, but the hot scheduler cannot find any hot peers in the three store. Because the store3 has 6k qps, but there are 3k regions in the hot table. In another word, maybe every region is only 2 qps.

store qps hot peer num hot table leader(without index)
1 9k 2 4239
2 7k 3 3405
3 6k 0 3332
4 3k 2 3131
5 3k 6 3049
... ... ... ...
96 ... ... ...
total ... 300 60000

20221014171706

We add a scatter region scheduler for only row table to solve the problem as a workaround.

In the future, I hope that PD can solve similar problems by self.

Describe the feature you'd like

  1. we should use more reasonable and unified threshold for hot peers, whatever policy is used. statistics, scheduler: unreasonable two sets of hotspot thresholds #5155

  2. For stores with high qps and less hot peers, we should report topn region to PD. The hot scheduler will solve them. But its speed may be slow, such as there is a store with 4k region that need to schedule.

  3. We use a clustering algorithm to identify similar table, and to balance them.

Teachability, Documentation, Adoption, Migration Strategy

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/feature-requestCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions