Skip to content

schedule: operator.check will meet datarace #8263

@lhy1024

Description

@lhy1024

Bug Report

What did you do?

run a server which built with -race

What did you expect to see?

no datarace

What did you see instead?

WARNING: DATA RACE
Read at 0x00c059c28fc8 by goroutine 2253766:
  github.com/tikv/pd/pkg/schedule/operator.(*Operator).Check()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator.go:382 +0x544
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).pollNeedDispatchRegion()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:262 +0x144
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).PushOperators()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:281 +0x2e
  github.com/tikv/pd/pkg/schedule.(*Coordinator).drivePushOperator()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:412 +0x224
  github.com/tikv/pd/pkg/schedule.(*Coordinator).Run.func5()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:491 +0x33

Previous write at 0x00c059c28fc8 by goroutine 37567:
  sync/atomic.CompareAndSwapInt64()
      /usr/local/go/src/runtime/race_amd64.s:316 +0xb
  sync/atomic.CompareAndSwapInt64()
      <autogenerated>:1 +0x1a
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).Dispatch()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:145 +0x104
  github.com/tikv/pd/server/cluster.(*RaftCluster).HandleRegionHeartbeat()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster_worker.go:66 +0x5c6  github.com/tikv/pd/server.(*GrpcServer).RegionHeartbeat()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/grpc_service.go:1296 +0x1c59
  github.com/pingcap/kvproto/pkg/pdpb._PD_RegionHeartbeat_Handler()
      /go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20240403065636-c699538f7aa1/pkg/pdpb/pdpb.pb.go:9777 +0xb8
  github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).StreamServerInterceptor.func4()
      /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 +0x15e
  go.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainStreamServer.func9.1()
      /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:83 +0xc3
  go.etcd.io/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1()
      /go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20240320135013-950cd5fbe6ca/etcdserver/api/v3rpc/interceptor.go:238 +0x6f0
  go.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainStreamServer.func9()
      /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:86 +0x218
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1687 +0x1f50
  google.golang.org/grpc.(*Server).handleStream()
      /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1801 +0x19f3
  google.golang.org/grpc.(*Server).serveStreams.func2.1()
      /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1027 +0x15c

Goroutine 2253766 (running) created at:
  github.com/tikv/pd/pkg/schedule.(*Coordinator).Run()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:491 +0x44a
  github.com/tikv/pd/pkg/schedule.(*Coordinator).RunUntilStop()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:455 +0x48
  github.com/tikv/pd/server/cluster.(*schedulingController).runCoordinator()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/scheduling_controller.go:121 +0xe6
  github.com/tikv/pd/server/cluster.(*schedulingController).startSchedulingJobs.func2()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/scheduling_controller.go:98 +0x33

Goroutine 37567 (running) created at:
  google.golang.org/grpc.(*Server).serveStreams.func2()
      /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1038 +0x224
  google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/pkg/mod/google.golang.org/grpc@v1.62.1/internal/transport/http2_server.go:619 +0x3a01
  google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/pkg/mod/google.golang.org/grpc@v1.62.1/internal/transport/http2_server.go:660 +0x285
  google.golang.org/grpc.(*Server).serveStreams()
      /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1021 +0x6bb
  google.golang.org/grpc.(*Server).handleRawConn.func1()
      /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:957 +0x86
==================

What version of PD are you using (pd-server -V)?

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-5.4This bug affects the 5.4.x(LTS) versions.affects-6.1This bug affects the 6.1.x(LTS) versions.affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.severity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    Status

    Closed

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions