Bug Report

What version of TiKV are you using?

./tikv-server -V
TiKV
Release Version: 8.5.0-alpha
Edition: Community
Git Commit Hash: 5c4a575faadb976616813028609550185afb5971
Git Commit Branch: HEAD
UTC Build Time: 2024-12-17 13:26:31
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
2024-12-18T10:42:05.840+0800

What operating system and CPU are you using?

8c/32g

Steps to reproduce

1、deploy pd ms
2、run tpcc
3、set placement rules
the first time：
[2024/12/18 09:49:53.688 +08:00] [INFO] [pdutil.go:591] ["placement show"]
[
{
"group_id": "pd",
"id": "rule0",
"start_key": "",
"end_key": "",
"role": "voter",
"is_witness": false,
"count": 2,
"label_constraints": [
{
"key": "zone",
"op": "in",
"values": [
"zone1"
]
}
],
"location_labels": [
"dc",
"zone",
"rack",
"host"
],
"create_timestamp": 1734486593
},
{
"group_id": "pd",
"id": "rule1",
"start_key": "",
"end_key": "",
"role": "learner",
"is_witness": false,
"count": 2,
"label_constraints": [
{
"key": "zone",
"op": "in",
"values": [
"zone2"
]
}
],
"location_labels": [
"dc",
"zone",
"rack",
"host"
],
"create_timestamp": 1734486593
},
{
"group_id": "pd",
"id": "rule2",
"start_key": "",
"end_key": "",
"role": "follower",
"is_witness": false,
"count": 1,
"label_constraints": [
{
"key": "zone",
"op": "in",
"values": [
"zone3"
]
}
],
"location_labels": [
"dc",
"zone",
"rack",
"host"
],
"create_timestamp": 1734486593
}
]

the second time
[2024/12/18 10:11:15.396 +08:00] [INFO] [pdutil.go:591] ["placement show"]
[
{
"group_id": "pd",
"id": "rule0",
"start_key": "",
"end_key": "",
"role": "voter",
"is_witness": false,
"count": 3,
"location_labels": [
"dc",
"zone",
"rack",
"host"
],
"version": 1,
"create_timestamp": 1734486593
}
]

4、placement rules changed at 2024/12/18 10:11:21 which was not done by case and we can the scheduling leader changed at this time
sh-5.1# ./pd-ctl config placement-rules show
[
{
"group_id": "pd",
"id": "default",
"start_key": "",
"end_key": "",
"role": "voter",
"is_witness": false,
"count": 3
},
{
"group_id": "pd",
"id": "rule0",
"start_key": "",
"end_key": "",
"role": "voter",
"is_witness": false,
"count": 3,
"location_labels": [
"dc",
"zone",
"rack",
"host"
],
"create_timestamp": 1734492070
}
]

5、scale in tikv from 6 to 5

What did you expect?

scale in can succeed

What did happened?

tikv scale in blocked due to the wrong placement rules