Skip to content

pd client panics with "tsoStream.recvLoop internal panic" #9051

@overvenus

Description

@overvenus

Bug Report

/opt/tidb/db.log:[2025/02/09 12:14:02.117 +00:00] [FATAL] [stream.go:330] ["tsoStream.recvLoop internal panic"] [stacktrace="github.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop.func1\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:330\nruntime.gopanic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.6.linux-amd64/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20241212030209-7e3ff8601a2a/global.go:54\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).checkMonotonicity\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:528\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).processRequests.func2\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:468\ngithub.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:444"] [panicMessage="[tso] timestamp fallback"] [stack="github.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop.func1\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:330\nruntime.gopanic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.6.linux-amd64/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20241212030209-7e3ff8601a2a/global.go:54\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).checkMonotonicity\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:528\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).processRequests.func2\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:468\ngithub.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:444"]

Test: https://tcms.pingcap.net/dashboard/executions/case/13077637

What did you do?

Run jepsen tests.

What did you expect to see?

No panic.

What version of PD are you using (pd-server -V)?

TiDB: c292ec642b3fd69904c4d5223415ba6a2a040a51
PD: debceaf

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.impact/panicseverity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions