non-default keyspace groups use the same timestamp path by mistake #6465
Description
Bug Report
What did you do?
Added TestGetMinTS test but it had random failure. The root cause is that non-default keyspace groups use the same timestamp path by mistake, which caused random failure when saving timestamp in etcd due to transaction conflict thus caused keyspace group primary to be reset unexpectedly.
What did you expect to see?
No random test failure in TestGetMinTS.
What did you see instead?
There were random failures due to the root cause described above.
2023-05-15T00:29:19.5492594Z [2023/05/15 00:29:19.548 +00:00] [WARN] [tso.go:289] ["clock offset"] [jet-lag=221.789286ms] [prev-physical=2023/05/15 00:29:19.326 +00:00] [now=2023/05/15 00:29:19.548 +00:00] [update-physical-interval=50ms]
2023-05-15T00:29:19.5600840Z [2023/05/15 00:29:19.559 +00:00] [WARN] [tso.go:289] ["clock offset"] [jet-lag=169.907035ms] [prev-physical=2023/05/15 00:29:19.389 +00:00] [now=2023/05/15 00:29:19.559 +00:00] [update-physical-interval=50ms]
2023-05-15T00:29:19.5651800Z [2023/05/15 00:29:19.564 +00:00] [WARN] [allocator_manager.go:752] ["failed to update allocator's timestamp"] [dc-location=global] [error="[PD:etcd:ErrEtcdTxnConflict]etcd transaction failed, conflicted and rolled back"]
2023-05-15T00:29:19.5668265Z [2023/05/15 00:29:19.566 +00:00] [INFO] [tso.go:375] ["reset the timestamp in memory"]
2023-05-15T00:29:19.5675787Z [2023/05/15 00:29:19.567 +00:00] [INFO] [lease.go:172] ["stop lease keep alive worker"] [purpose="keyspace group primary election"]
2023-05-15T00:29:19.5726002Z [2023/05/15 00:29:19.572 +00:00] [INFO] [leadership.go:211] ["current leadership is deleted"] [leader-key=/ms/7233199744956437750/tso/00001/primary] [purpose="keyspace group primary election"]
2023-05-15T00:29:19.5733101Z [2023/05/15 00:29:19.573 +00:00] [INFO] [leadership.go:211] ["current leadership is deleted"] [leader-key=/ms/7233199744956437750/tso/00001/primary] [purpose="keyspace group primary election"]
2023-05-15T00:29:19.5746233Z [2023/05/15 00:29:19.574 +00:00] [INFO] [global_allocator.go:499] ["the tso primary has changed, try to re-campaign a primary"]
2023-05-15T00:29:19.5750728Z [2023/05/15 00:29:19.574 +00:00] [INFO] [global_allocator.go:507] ["start to campaign the primary"] [campaign-tso-primary-name=http://127.0.0.1:44687-00001]