statistics: add more write query kind #10786

lhy1024 · 2021-08-19T18:34:40Z

What problem does this PR solve?

Issue Number: close #10507

Problem Summary:

What is changed and how it works?

it collect quert stats in scheduler and flush them in tick to pd.rs

but it can collect raw write query again, so we skip some tests.

Related changes

Need to cherry-pick to the release branch

Check List

Tests

Unit test

insert 1200 tables

write test with Pessimistic

Release note

None.

ti-chi-bot · 2021-08-19T18:34:41Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

5kbpers
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 · 2021-08-19T19:17:16Z

/rebuild

nolouch · 2021-08-20T05:04:01Z

cc @JmPotato

components/raftstore/src/store/worker/pd.rs

Signed-off-by: lhy1024 <admin@liudos.us>

nolouch · 2021-08-20T11:05:42Z

/rebuild

nolouch

lgtm

5kbpers · 2021-08-23T03:22:00Z

components/raftstore/src/store/worker/pd.rs

            if read_bytes < hotspot_byte_report_threshold()
                && read_keys < hotspot_key_report_threshold()
-                && read_query_stats.get_read_query_num() < hotspot_query_num_report_threshold()
+                && query_stats.get_read_query_num() < hotspot_query_num_report_threshold()


Should we add a condition for write queries here?

Currently, the read hotspot information is obtained from the store heartbeat, and the write hotspot information is obtained from the region heartbeat, and the corresponding threshold will be added when all the hotspot information is done in the store heartbeat later.

comment

Yisaer

basically lgtm

Yisaer · 2021-08-23T04:18:06Z

src/read_pool.rs

    impl FlowStatsReporter for DummyReporter {
        fn report_read_stats(&self, _read_stats: ReadStats) {}
+        fn report_write_stats(&self, _write_stats: WriteStats) {}


I noticed that FlowStatsReporter will report write stats while the file was still named as read_pool.rs.

I did not find a more suitable place, do you have any good suggestions

ti-chi-bot · 2021-08-23T04:18:17Z

@Yisaer: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

basically lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

lhy1024 · 2021-08-23T08:44:15Z

@5kbpers @Yisaer PTAL

ti-chi-bot · 2021-08-24T03:05:04Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-08-24T03:05:06Z

@nolouch: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

lhy1024 · 2021-08-24T03:44:18Z

/merge

ti-chi-bot · 2021-08-24T03:44:18Z

@lhy1024: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-08-24T03:44:19Z

@lhy1024: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

5kbpers · 2021-08-24T04:00:09Z

/merge

ti-chi-bot · 2021-08-24T04:00:10Z

@5kbpers: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-08-24T04:00:12Z

This pull request has been accepted and is ready to merge.

Commit hash: 1e4a234

lhy1024 · 2021-08-24T07:29:04Z

/rebuild

Signed-off-by: lhy1024 <admin@liudos.us>

JmPotato · 2021-08-24T09:58:02Z

/merge

ti-chi-bot · 2021-08-24T09:58:03Z

@JmPotato: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-08-24T09:58:04Z

@JmPotato: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

5kbpers · 2021-08-24T10:05:11Z

/merge

ti-chi-bot · 2021-08-24T10:05:12Z

@5kbpers: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-08-24T10:05:14Z

This pull request has been accepted and is ready to merge.

Commit hash: 57fd8aa

nolouch · 2021-08-24T10:18:50Z

/rebuild

nolouch · 2021-08-24T10:18:56Z

/test

lhy1024 · 2021-08-24T10:30:06Z

/rebuild

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-08-24T10:58:25Z

cherry pick to release-5.2 in PR #10809

* cherry pick #10786 to release-5.2 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * fix conflict Signed-off-by: lhy1024 <admin@liudos.us> * fix lint Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: lhy1024 <admin@liudos.us> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 19, 2021

lhy1024 force-pushed the write-qps branch from b182f35 to 0f77037 Compare August 19, 2021 18:45

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 19, 2021

lhy1024 force-pushed the write-qps branch from 0f77037 to 36efbb4 Compare August 19, 2021 18:49

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 19, 2021

update

f2164e8

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the write-qps branch from 36efbb4 to f2164e8 Compare August 19, 2021 19:00

lhy1024 requested review from 5kbpers and nolouch August 19, 2021 19:00

lhy1024 added the needs-cherry-pick-release-5.2 Type: Need cherry pick to release-5.2 label Aug 19, 2021

lhy1024 mentioned this pull request Aug 20, 2021

Adjustable Priority Multidimensional Hot Scheduler Tasklist tikv/pd#3869

Closed

28 tasks

nolouch reviewed Aug 20, 2021

View reviewed changes

components/raftstore/src/store/worker/pd.rs Show resolved Hide resolved

components/raftstore/src/store/worker/pd.rs Outdated Show resolved Hide resolved

address comments

1e4a234

Signed-off-by: lhy1024 <admin@liudos.us>

nolouch approved these changes Aug 20, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 20, 2021

5kbpers previously approved these changes Aug 23, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 23, 2021

5kbpers reviewed Aug 23, 2021

View reviewed changes

Yisaer approved these changes Aug 23, 2021

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 24, 2021

Merge branch 'master' into write-qps

d780f30

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 24, 2021

Merge branch 'master' of http://github.com/tikv/tikv into write-qps

57fd8aa

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot removed status/can-merge Indicates a PR has been approved by a committer. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 24, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 24, 2021

ti-chi-bot merged commit 7625317 into tikv:master Aug 24, 2021

ti-srebot pushed a commit to ti-srebot/tikv that referenced this pull request Aug 24, 2021

cherry pick tikv#10786 to release-5.2

bd642fd

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Aug 24, 2021

statistics: add more write query kind (#10786) #10809

Merged

statistics: add more write query kind #10786

statistics: add more write query kind #10786

Uh oh!

Conversation

lhy1024 commented Aug 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

ti-chi-bot commented Aug 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lhy1024 commented Aug 19, 2021

Uh oh!

nolouch commented Aug 20, 2021

Uh oh!

Uh oh!

Uh oh!

nolouch commented Aug 20, 2021

Uh oh!

nolouch left a comment

Choose a reason for hiding this comment

Uh oh!

5kbpers Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

lhy1024 Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

Yisaer left a comment

Choose a reason for hiding this comment

Uh oh!

Yisaer Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

lhy1024 Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot commented Aug 23, 2021

Uh oh!

lhy1024 commented Aug 23, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

lhy1024 commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

5kbpers commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

lhy1024 commented Aug 24, 2021

Uh oh!

JmPotato commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

5kbpers commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

ti-chi-bot commented Aug 24, 2021

Uh oh!

nolouch commented Aug 24, 2021

Uh oh!

nolouch commented Aug 24, 2021

Uh oh!

lhy1024 commented Aug 24, 2021

lhy1024 commented Aug 19, 2021 •

edited

Loading

ti-chi-bot commented Aug 19, 2021 •

edited

Loading