Skip to content

scheduler: move hot region should be splitted if it's size is too large . (#6281) #6371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 27, 2023
Merged

scheduler: move hot region should be splitted if it's size is too large . (#6281) #6371

merged 2 commits into from
Apr 27, 2023

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #6281

What problem does this PR solve?

Issue Number: Ref #6328,Close tikv/tikv#14333, ref #tikv/tikv#14214, ref tikv/tikv#14135

What is changed and how does it work?

If the moved region's size is bigger than the max-movable-hot-peer-size, it will be splited by hot buckets.
It will brings two benefits:

  1. bucket split will not be always happened, so the total region count is reasonable
  2. bucket split is on time, not depends on split-bucket-scheduler

The split-bucket-scheduler will be removed from the default list

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

This PR:
img_v2_76d62145-dc58-497c-b07f-51cdc03d96fg

master:
img_v2_e3e332eb-b7e3-4668-a2ea-49d61716482g

Code changes

Side effects

Related changes

Release note

None.

@ti-chi-bot ti-chi-bot mentioned this pull request Apr 24, 2023
Merged
@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. type/cherry-pick-for-release-7.1 This PR is cherry-picked to release-7.1 from a source PR. labels Apr 24, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 24, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • bufferflies
  • lhy1024

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 24, 2023
@bufferflies
scheduler: move hot region should be splitted if it's size is too lar…
8a03d9f 
…ge . (tikv#6281)

* add createSplitOperator

Signed-off-by: bufferflies <1045931706@qq.com>

* consider string nil

Signed-off-by: bufferflies <1045931706@qq.com>

* add unit test for split buckets

Signed-off-by: bufferflies <1045931706@qq.com>

* enable split in hot region

Signed-off-by: bufferflies <1045931706@qq.com>

* lint

Signed-off-by: bufferflies <1045931706@qq.com>

* add stringer for resourceTy

Signed-off-by: bufferflies <1045931706@qq.com>

* remove region keys and hot split keys in additional info

Signed-off-by: bufferflies <1045931706@qq.com>

* pass unit test

Signed-off-by: bufferflies <1045931706@qq.com>

* address comments

Signed-off-by: bufferflies <1045931706@qq.com>

* remove one case from TestLoadKeyspaceGroupsAssignment

Signed-off-by: bufferflies <1045931706@qq.com>

* revert TestLoadKeyspaceGroupsAssignment

Signed-off-by: bufferflies <1045931706@qq.com>

---------

Signed-off-by: bufferflies <1045931706@qq.com>
Co-authored-by: Ryan Leung <rleungx@gmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

fix flaky key for TestLoadKeyspaceGroupsAssignment

Signed-off-by: bufferflies <1045931706@qq.com>
@bufferflies bufferflies force-pushed the cherry-pick-6281-to-release-7.1 branch from d23ad1a to 8a03d9f Compare April 24, 2023 04:16
@bufferflies bufferflies requested a review from lhy1024 April 24, 2023 04:19
@ti-chi-bot ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 24, 2023
@codecov
Copy link

codecov bot commented Apr 24, 2023 •
edited
Loading

Codecov Report

Patch coverage: 57.89% and project coverage change: -0.20 ⚠️

Comparison is base (d0b04d4) 75.05% compared to head (43016e4) 74.86%.

Additional details and impacted files 
@@               Coverage Diff               @@
##           release-7.1    #6371      +/-   ##
===============================================
- Coverage        75.05%   74.86%   -0.20%     
===============================================
  Files              408      408              
  Lines            40481    40546      +65     
===============================================
- Hits             30384    30354      -30     
- Misses            7445     7532      +87     
- Partials          2652     2660       +8
Flag Coverage Δ
unittests 74.86% <57.89%> (-0.20%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
server/cluster/cluster.go 81.28% <0.00%> (-0.56%) ⬇️
server/config/config.go 75.76% <ø> (ø)
pkg/schedule/schedulers/hot_region.go 82.45% <44.64%> (-2.04%) ⬇️
pkg/mock/mockcluster/mockcluster.go 94.71% <100.00%> (+0.45%) ⬆️
pkg/schedule/operator/operator.go 92.85% <100.00%> (+0.04%) ⬆️
pkg/statistics/buckets/bucket_stat_informer.go 94.28% <100.00%> (ø)
pkg/statistics/buckets/hot_bucket_cache.go 90.74% <100.00%> (+0.54%) ⬆️
pkg/statistics/buckets/hot_bucket_task.go 86.36% <100.00%> (+0.64%) ⬆️
pkg/utils/keyutil/util.go 100.00% <100.00%> (ø)

... and 39 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@bufferflies
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 24, 2023

@bufferflies: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 24, 2023

This pull request has been accepted and is ready to merge.

Commit hash: 43016e4

@ti-chi-bot ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 24, 2023
@VelocityLight VelocityLight added the cherry-pick-approved Cherry pick PR approved by release team. label Apr 27, 2023
@bufferflies
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 27, 2023

@bufferflies: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot bot merged commit 387d8bb into tikv:release-7.1 Apr 27, 2023
@VelocityLight VelocityLight removed the cherry-pick-approved Cherry pick PR approved by release team. label Jun 8, 2023
@VelocityLight VelocityLight added the cherry-pick-approved Cherry pick PR approved by release team. label Jun 8, 2023
@ti-chi-bot ti-chi-bot removed the cherry-pick-approved Cherry pick PR approved by release team. label Jul 24, 2023
@ti-chi-bot ti-chi-bot added the cherry-pick-approved Cherry pick PR approved by release team. label Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lhy1024 lhy1024 lhy1024 approved these changes

@bufferflies bufferflies bufferflies approved these changes

Assignees
No one assigned
Labels
cherry-pick-approved Cherry pick PR approved by release team. release-note-none Denotes a PR that doesn't merit a release note. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/cherry-pick-for-release-7.1 This PR is cherry-picked to release-7.1 from a source PR.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ti-chi-bot @bufferflies @lhy1024 @VelocityLight