Skip to content

mcs, tso: timestamp fallback after merging #6686

@binshi-bing

Description

@binshi-bing

Enhancement Task

// split keyspace 1 from group 0 to group 1
root@serverless-cluster-pd-0:/# ./pd-ctl keyspace-group split 0 1 1
Success!
root@serverless-cluster-pd-0:/# ./pd-ctl keyspace-group merge 0 1
Failed! [500] "keyspace group is in split state"

// Send traffic to keyspace 1 to trigger finish split for group 1
root@tso-bench:/# ./pd-tso-bench -v -duration 250000s -pd "http://serverless-cluster-pd-0.serverless-cluster-pd-peer.tidb-serverless.svc:2379" -client 10 -c 3 -interval 30s -keyspace 1

// Now we can merge 1 to 0
root@serverless-cluster-pd-0:/# ./pd-ctl keyspace-group merge 0 1
Success!

// But pd-tso-bench reported time fallback immediately after above merging.
panic: global timestamp fallback, new ts (1687812834954, 1003) <= the last one (1687812834975, 50). keyspace: 1, keyspace group: 0

goroutine 693 [running]:
github.com/tikv/pd/client.(*tsoClient).compareAndSwapTS(0x4000439d40, {0x93f24b, 0x6}, 0x188f97c068a, 0x3eb, 0x0, 0x2)
/go/src/github.com/tikv/pd/client/tso_dispatcher.go:742 +0x244
github.com/tikv/pd/client.(*tsoClient).processRequests(0x4000439d40, {0xa6e3e0, 0x4000f94bd0}, {0x93f24b, 0x6}, 0x40000c50e0, {0x0?, 0x0, 0x4000160280?})
/go/src/github.com/tikv/pd/client/tso_dispatcher.go:720 +0x250
github.com/tikv/pd/client.(*tsoClient).handleDispatcher(0x4000439d40, {0xa79090?, 0x4000538040}, {0x93f24b, 0x6}, 0x40000c50e0)
/go/src/github.com/tikv/pd/client/tso_dispatcher.go:445 +0xde4
created by github.com/tikv/pd/client.(*tsoClient).createTSODispatcher
/go/src/github.com/tikv/pd/client/tso_dispatcher.go:290 +0x264

Here is the PD client log.
https://gist.githubusercontent.com/binshi-bing/68fadbf399dc5991e9c65341d7375636/raw/05de99538b8a5dc9d65ace1ddf8462928541c05a/gistfile1.txt

All Client/TSO log can be found https://gist.github.com/binshi-bing/68fadbf399dc5991e9c65341d7375636/edit

Metadata

Metadata

Assignees

Labels

type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions