-
Notifications
You must be signed in to change notification settings - Fork 742
Closed
Labels
affects-8.5This bug affects the 8.5.x(LTS) versions.This bug affects the 8.5.x(LTS) versions.impact/panicseverity/majortype/bugThe issue is confirmed as a bug.The issue is confirmed as a bug.
Description
Bug Report
store/TiDB master bank-multitable auto-retry auto-retry-limit :default select FOR UPDATE txn-mode pessimistic isolation :repeatable-read nemesis kill,partition-one,schedules,stop/20241119T091111.000Z/no
de-0.node-peer.jepsen-tps-7527309-1-659/db.log:[2024/11/19 09:22:35.584 +00:00] [FATAL] [tso_stream.go:330] ["tsoStream.recvLoop internal panic"] [stacktrace="[github.com/tikv/pd/client.(*tsoStream).recv](http://github.com/tikv/pd/client.(*tsoStream).recv)
Loop.func1\n\t/root/go/pkg/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:330\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWri](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:330/nruntime.gopanic/n/t/usr/local/go/src/runtime/panic.go:785/ngo.uber.org/zap/zapcore.CheckWri)
teAction.OnWrite\n\t/root/go/pkg/mod/[go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber](http://go.uber.org/zap@v1.27.0/zapcore/entry.go:196/ngo.uber.org/zap/zapcore.(*CheckedEntry).Write/n/t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262/ngo.uber)
.org/zap.(*Logger).Panic\n\t/root/go/pkg/mod/[go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20240314023424-862ccc32f18d/global.go](http://go.uber.org/zap@v1.27.0/logger.go:285/ngithub.com/pingcap/log.Panic/n/t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20240314023424-862ccc32f18d/global.go)
:54\[ngithub.com/tikv/pd/client.(*tsoDispatcher).checkMonotonicity\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:647\ngithub.com/tikv/pd/client](http://ngithub.com/tikv/pd/client.(*tsoDispatcher).checkMonotonicity/n/t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:647/ngithub.com/tikv/pd/client).(*tsoD
ispatcher).processRequests.func2\n\t/root/go/pkg/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:597\ngithub.com/tikv/pd/client.(*tsoStream).recvLoop\n\t/root/go/pkg/m](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:597/ngithub.com/tikv/pd/client.(*tsoStream).recvLoop/n/t/root/go/pkg/m)
od/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:444](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:444)"] [panicMessage="[tso] timestamp fallback"] [stack="[github.com/tikv/pd/client.(*tsoStream).recvLoop.func1\n\t/root/go/pk](http://github.com/tikv/pd/client.(*tsoStream).recvLoop.func1/n/t/root/go/pk)
g/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:330\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/root](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:330/nruntime.gopanic/n/t/usr/local/go/src/runtime/panic.go:785/ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite/n/t/root)
/go/pkg/mod/[go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic](http://go.uber.org/zap@v1.27.0/zapcore/entry.go:196/ngo.uber.org/zap/zapcore.(*CheckedEntry).Write/n/t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262/ngo.uber.org/zap.(*Logger).Panic)\
n\t/root/go/pkg/mod/[go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20240314023424-862ccc32f18d/global.go:54\ngithub.com/tikv/pd/c](http://go.uber.org/zap@v1.27.0/logger.go:285/ngithub.com/pingcap/log.Panic/n/t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20240314023424-862ccc32f18d/global.go:54/ngithub.com/tikv/pd/c)
lient.(*tsoDispatcher).checkMonotonicity\n\t/root/go/pkg/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:647\ngithub.com/tikv/pd/client.(*tsoDispatcher).processRequest](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:647/ngithub.com/tikv/pd/client.(*tsoDispatcher).processRequest)
s.func2\n\t/root/go/pkg/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:597\ngithub.com/tikv/pd/client.(*tsoStream).recvLoop\n\t/root/go/pkg/mod/github.com/tikv/pd/cli](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:597/ngithub.com/tikv/pd/client.(*tsoStream).recvLoop/n/t/root/go/pkg/mod/github.com/tikv/pd/cli)
ent@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:444"]
Test: https://tcms.pingcap.net/dashboard/executions/plan/7527309
The issue was found by @SpadeA-Tang, thank you!
What did you do?
Run jepsen tests.
What did you expect to see?
No panic.
What version of PD are you using (pd-server -V
)?
TiDB: pingcap/tidb@5448f9f
PD: 60b960a
Metadata
Metadata
Assignees
Labels
affects-8.5This bug affects the 8.5.x(LTS) versions.This bug affects the 8.5.x(LTS) versions.impact/panicseverity/majortype/bugThe issue is confirmed as a bug.The issue is confirmed as a bug.