-
Notifications
You must be signed in to change notification settings - Fork 742
Closed
Labels
affects-6.5This bug affects the 6.5.x(LTS) versions.This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.This bug affects the 8.5.x(LTS) versions.impact/panicseverity/majortype/bugThe issue is confirmed as a bug.The issue is confirmed as a bug.
Description
Bug Report
/opt/tidb/db.log:[2025/02/09 12:14:02.117 +00:00] [FATAL] [stream.go:330] ["tsoStream.recvLoop internal panic"] [stacktrace="github.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop.func1\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:330\nruntime.gopanic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.6.linux-amd64/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20241212030209-7e3ff8601a2a/global.go:54\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).checkMonotonicity\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:528\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).processRequests.func2\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:468\ngithub.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:444"] [panicMessage="[tso] timestamp fallback"] [stack="github.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop.func1\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:330\nruntime.gopanic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.6.linux-amd64/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20241212030209-7e3ff8601a2a/global.go:54\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).checkMonotonicity\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:528\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).processRequests.func2\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:468\ngithub.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:444"]
Test: https://tcms.pingcap.net/dashboard/executions/case/13077637
What did you do?
Run jepsen tests.
What did you expect to see?
No panic.
What version of PD are you using (pd-server -V
)?
TiDB: c292ec642b3fd69904c4d5223415ba6a2a040a51
PD: debceaf
Metadata
Metadata
Assignees
Labels
affects-6.5This bug affects the 6.5.x(LTS) versions.This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.This bug affects the 8.5.x(LTS) versions.impact/panicseverity/majortype/bugThe issue is confirmed as a bug.The issue is confirmed as a bug.