timestamp fallback in default keyspace group #6698
Description
Bug Report
What did you do?
only stop and restart server kubectl delete pod serverless-cluster-pd-0 serverless-cluster-pd-1 serverless-cluster-pd-2 -n tidb-serverless --force --grace-period=0 && kubectl delete po pd-tso-server-0 pd-tso-server-1 -n tidb-serverless --force --grace-period=0
What did you expect to see?
normal
What did you see instead?
[2023/06/27 09:16:58.476 +00:00] [INFO] [tso_dispatcher.go:746] ["[tso] keyspace group changed"] [dc-location=global] [old-group-id=1] [new-group-id=0]
[2023/06/27 09:16:58.476 +00:00] [INFO] [tso_dispatcher.go:313] ["[tso] exit tso dispatcher"] [dc-location=global]
panic: global timestamp fallback, new ts (1687857418453, 1) <= the last one (1687857418453, 57). last keyspace group: 1, keyspace in request: 0, keyspace group in request: 1, keyspace group in response: 0
goroutine 71 [running]:
github.com/tikv/pd/client.(*tsoClient).compareAndSwapTS(0x4000000120, {0x93f68b, 0x6}, 0x0, 0x188fc2450d5, 0x1, 0x0, 0x1)
/go/src/github.com/tikv/pd/client/tso_dispatcher.go:756 +0x454
github.com/tikv/pd/client.(*tsoClient).processRequests(0x4000000120, {0xa6ea40, 0x4000cfb210}, {0x93f68b, 0x6}, 0x40004fa050, {0x0?, 0x0, 0x400099ee40?})
/go/src/github.com/tikv/pd/client/tso_dispatcher.go:721 +0x254
github.com/tikv/pd/client.(*tsoClient).handleDispatcher(0x4000000120, {0xa796f0?, 0x4000265600}, {0x93f68b, 0x6}, 0x40004fa050)
/go/src/github.com/tikv/pd/client/tso_dispatcher.go:446 +0xde4
created by github.com/tikv/pd/client.(*tsoClient).createTSODispatcher
/go/src/github.com/tikv/pd/client/tso_dispatcher.go:291 +0x264