Skip to content

pd client panics with "[tso] timestamp fallback" #8836

@overvenus

Description

@overvenus

Bug Report

store/TiDB master bank-multitable auto-retry auto-retry-limit :default select FOR UPDATE txn-mode pessimistic isolation :repeatable-read nemesis kill,partition-one,schedules,stop/20241119T091111.000Z/no
de-0.node-peer.jepsen-tps-7527309-1-659/db.log:[2024/11/19 09:22:35.584 +00:00] [FATAL] [tso_stream.go:330] ["tsoStream.recvLoop internal panic"] [stacktrace="[github.com/tikv/pd/client.(*tsoStream).recv](http://github.com/tikv/pd/client.(*tsoStream).recv)
Loop.func1\n\t/root/go/pkg/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:330\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWri](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:330/nruntime.gopanic/n/t/usr/local/go/src/runtime/panic.go:785/ngo.uber.org/zap/zapcore.CheckWri)
teAction.OnWrite\n\t/root/go/pkg/mod/[go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber](http://go.uber.org/zap@v1.27.0/zapcore/entry.go:196/ngo.uber.org/zap/zapcore.(*CheckedEntry).Write/n/t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262/ngo.uber)
.org/zap.(*Logger).Panic\n\t/root/go/pkg/mod/[go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20240314023424-862ccc32f18d/global.go](http://go.uber.org/zap@v1.27.0/logger.go:285/ngithub.com/pingcap/log.Panic/n/t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20240314023424-862ccc32f18d/global.go)
:54\[ngithub.com/tikv/pd/client.(*tsoDispatcher).checkMonotonicity\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:647\ngithub.com/tikv/pd/client](http://ngithub.com/tikv/pd/client.(*tsoDispatcher).checkMonotonicity/n/t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:647/ngithub.com/tikv/pd/client).(*tsoD
ispatcher).processRequests.func2\n\t/root/go/pkg/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:597\ngithub.com/tikv/pd/client.(*tsoStream).recvLoop\n\t/root/go/pkg/m](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:597/ngithub.com/tikv/pd/client.(*tsoStream).recvLoop/n/t/root/go/pkg/m)
od/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:444](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:444)"] [panicMessage="[tso] timestamp fallback"] [stack="[github.com/tikv/pd/client.(*tsoStream).recvLoop.func1\n\t/root/go/pk](http://github.com/tikv/pd/client.(*tsoStream).recvLoop.func1/n/t/root/go/pk)
g/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:330\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/root](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:330/nruntime.gopanic/n/t/usr/local/go/src/runtime/panic.go:785/ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite/n/t/root)
/go/pkg/mod/[go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic](http://go.uber.org/zap@v1.27.0/zapcore/entry.go:196/ngo.uber.org/zap/zapcore.(*CheckedEntry).Write/n/t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262/ngo.uber.org/zap.(*Logger).Panic)\
n\t/root/go/pkg/mod/[go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20240314023424-862ccc32f18d/global.go:54\ngithub.com/tikv/pd/c](http://go.uber.org/zap@v1.27.0/logger.go:285/ngithub.com/pingcap/log.Panic/n/t/root/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20240314023424-862ccc32f18d/global.go:54/ngithub.com/tikv/pd/c)
lient.(*tsoDispatcher).checkMonotonicity\n\t/root/go/pkg/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:647\ngithub.com/tikv/pd/client.(*tsoDispatcher).processRequest](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:647/ngithub.com/tikv/pd/client.(*tsoDispatcher).processRequest)
s.func2\n\t/root/go/pkg/mod/[github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:597\ngithub.com/tikv/pd/client.(*tsoStream).recvLoop\n\t/root/go/pkg/mod/github.com/tikv/pd/cli](http://github.com/tikv/pd/client@v0.0.0-20241016064947-b70107ec31e6/tso_dispatcher.go:597/ngithub.com/tikv/pd/client.(*tsoStream).recvLoop/n/t/root/go/pkg/mod/github.com/tikv/pd/cli)
ent@v0.0.0-20241016064947-b70107ec31e6/tso_stream.go:444"]

Test: https://tcms.pingcap.net/dashboard/executions/plan/7527309

The issue was found by @SpadeA-Tang, thank you!

What did you do?

Run jepsen tests.

What did you expect to see?

No panic.

What version of PD are you using (pd-server -V)?

TiDB: pingcap/tidb@5448f9f
PD: 60b960a

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions