Skip to content

flaky/data-race test TestPiTRAndFailureRestore/TestOperator/TestFailure #59094

New issue
New issue
Open
tikv/tikv
#18163
#59210
Open
flaky/data-race test TestPiTRAndFailureRestore/TestOperator/TestFailure#59094
tikv/tikv
#18163
#59210
Labels
component/brThis issue is related to BR of TiDB.severity/moderatetype/bugThe issue is confirmed as a bug.
@D3Hunter

Description

@D3Hunter
D3Hunter
opened on Jan 21, 2025

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

see https://do.pingcap.net/jenkins/blue/rest/organizations/jenkins/pipelines/pingcap/pipelines/tidb/pipelines/ghpr_check2/runs/26281/nodes/103/steps/575/log/?start=0
for TestOperator/TestFailure

=== RUN   TestOperator
    operator_test.go:80: 
        	Error Trace:	tests/realtikvtest/brietest/operator_test.go:80
        	            				tests/realtikvtest/brietest/operator_test.go:195
        	Error:      	the service gc safepoint exists
        	Test:       	TestOperator
        	Messages:   	it is struct { ServiceID string "json:\"service_id\""; ExpiredAt int64 "json:\"expired_at\""; SafePoint int64 "json:\"safe_point\"" }{ServiceID:"backup-stream-TestPiTRAndBackupInSQL-1", ExpiredAt:1737450741, SafePoint:455460398471053328}
--- FAIL: TestOperator (0.00s)
=== RUN   TestFailure
    operator_test.go:80: 
        	Error Trace:	tests/realtikvtest/brietest/operator_test.go:80
        	            				tests/realtikvtest/brietest/operator_test.go:248
        	Error:      	the service gc safepoint exists
        	Test:       	TestFailure
        	Messages:   	it is struct { ServiceID string "json:\"service_id\""; ExpiredAt int64 "json:\"expired_at\""; SafePoint int64 "json:\"safe_point\"" }{ServiceID:"backup-stream-TestPiTRAndBackupInSQL-1", ExpiredAt:1737450741, SafePoint:455460398471053328}
--- FAIL: TestFailure (0.00s)

for TestPiTRAndFailureRestore

WARNING: DATA RACE
Read at 0x00c056d155b0 by goroutine 243296:
  github.com/pingcap/tidb/pkg/domain.(*schemaValidator).IsLeaseExpired()
      pkg/domain/schema_validator.go:175 +0x4b
  github.com/pingcap/tidb/pkg/domain.(*Domain).IsLeaseExpired()
      pkg/domain/domain.go:1568 +0x6f
  github.com/pingcap/tidb/br/pkg/task.waitUntilSchemaReload.func1()
      br/pkg/task/stream.go:2065 +0x74
  github.com/pingcap/tidb/br/pkg/utils.WaitUntil()
      br/pkg/utils/wait.go:26 +0x67
  github.com/pingcap/tidb/br/pkg/task.waitUntilSchemaReload()
      br/pkg/task/stream.go:2067 +0xe4
  github.com/pingcap/tidb/br/pkg/task.restoreStream()
      br/pkg/task/stream.go:1644 +0x323a
  github.com/pingcap/tidb/br/pkg/task.RunStreamRestore()
      br/pkg/task/stream.go:1352 +0x163a
  github.com/pingcap/tidb/br/pkg/task.RunRestore()
      br/pkg/task/restore.go:722 +0x606
  tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore.(*LogBackupKit).RunStreamRestore.func11()
      tests/realtikvtest/brietest/pitr_test.go:190 +0x37e
  tests/realtikvtest/brietest/brietest_test.(*LogBackupKit).runAndCheck()
      tests/realtikvtest/brietest/pitr_test.go:279 +0x9e
  tests/realtikvtest/brietest/brietest_test.(*LogBackupKit).RunStreamRestore()
      tests/realtikvtest/brietest/pitr_test.go:181 +0x5d4
  tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore()
      tests/realtikvtest/brietest/pitr_test.go:605 +0x50a
  github.com/pingcap/failpoint.parseTerm()
      external/com_github_pingcap_failpoint/terms.go:149 +0x233
  github.com/pingcap/failpoint.parse()
      external/com_github_pingcap_failpoint/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      external/com_github_pingcap_failpoint/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).EnableCall()
      external/com_github_pingcap_failpoint/failpoint.go:94 +0x124
  github.com/pingcap/failpoint.(*Failpoints).EnableCall()
      external/com_github_pingcap_failpoint/failpoints.go:150 +0x296
  github.com/pingcap/failpoint.EnableCall()
      external/com_github_pingcap_failpoint/failpoints.go:270 +0x31b
  tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore()
      tests/realtikvtest/brietest/pitr_test.go:590 +0x31c
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1333 +0x12e
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func619()
      pkg/sessionctx/variable/sysvar.go:3309 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).Validate()
      pkg/sessionctx/variable/variable.go:317 +0x12b
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1330 +0xd2
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).setSysVariable()
      pkg/executor/set.go:155 +0x69d
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).Next()
      pkg/executor/set.go:105 +0xee6
  github.com/pingcap/tidb/pkg/executor/internal/exec.Next()
      pkg/executor/internal/exec/executor.go:460 +0x3a8
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next()
      pkg/executor/adapter.go:1268 +0x84
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor()
      pkg/executor/adapter.go:1017 +0x624
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay()
      pkg/executor/adapter.go:850 +0x3e7
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec()
      pkg/executor/adapter.go:613 +0x180c
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2290 +0x62b
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2152 +0x1d09
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext()
      pkg/testkit/testkit.go:425 +0x1030
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext()
      pkg/testkit/testkit.go:159 +0xab
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec()
      pkg/testkit/testkit.go:154 +0x117
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).RefreshSession()
      pkg/testkit/testkit.go:126 +0x3b2
  github.com/pingcap/tidb/pkg/testkit.NewTestKit()
      pkg/testkit/testkit.go:86 +0x4dd
  tests/realtikvtest/brietest/brietest_test.initTestKit()
      tests/realtikvtest/brietest/backup_restore_test.go:42 +0x164
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1342 +0x304
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).setSysVariable()
      pkg/executor/set.go:155 +0x69d
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).Next()
      pkg/executor/set.go:105 +0xee6
  github.com/pingcap/tidb/pkg/executor/internal/exec.Next()
      pkg/executor/internal/exec/executor.go:460 +0x3a8
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next()
      pkg/executor/adapter.go:1268 +0x84
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor()
      pkg/executor/adapter.go:1017 +0x624
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay()
      pkg/executor/adapter.go:850 +0x3e7
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec()
      pkg/executor/adapter.go:613 +0x180c
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2290 +0x62b
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2152 +0x1d09
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext()
      pkg/testkit/testkit.go:425 +0x1030
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext()
      pkg/testkit/testkit.go:159 +0xab
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec()
      pkg/testkit/testkit.go:154 +0x117
  github.com/pingcap/tidb/tests/realtikvtest.CreateMockStoreAndDomainAndSetup()
      tests/realtikvtest/testkit.go:146 +0x464
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func217()
      pkg/sessionctx/variable/sysvar.go:1147 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).ValidateWithRelaxedValidation()
      pkg/sessionctx/variable/variable.go:377 +0x242
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:142 +0xa24
  fmt.Sscanf()
      GOROOT/src/fmt/scan.go:114 +0x18e
  github.com/pingcap/tidb/pkg/sessionctx/variable.parseByteSize()
      pkg/sessionctx/variable/varsutil.go:406 +0x1d
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func261()
      pkg/sessionctx/variable/sysvar.go:1418 +0x44
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:143 +0xa94
  github.com/pingcap/tidb/pkg/sessionctx/variable.parseSchemaCacheSize()
      pkg/sessionctx/variable/varsutil.go:638 +0x17c
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func620()
      pkg/sessionctx/variable/sysvar.go:3317 +0x67
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:143 +0xa94
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func619()
      pkg/sessionctx/variable/sysvar.go:3309 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).ValidateWithRelaxedValidation()
      pkg/sessionctx/variable/variable.go:377 +0x242
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:142 +0xa24
  github.com/pingcap/tidb/pkg/domain.(*Domain).LoadSysVarCacheLoop()
      pkg/domain/domain.go:1961 +0x93
  github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl()
      pkg/session/session.go:3538 +0x864
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func218()
      pkg/sessionctx/variable/sysvar.go:1154 +0x56
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:143 +0xa94
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func217()
      pkg/sessionctx/variable/sysvar.go:1147 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).ValidateWithRelaxedValidation()
      pkg/sessionctx/variable/variable.go:377 +0x242
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:142 +0xa24
  fmt.Sscanf()
      GOROOT/src/fmt/scan.go:114 +0x18e
  github.com/pingcap/tidb/pkg/sessionctx/variable.parseByteSize()
      pkg/sessionctx/variable/varsutil.go:406 +0x1d
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func261()
      pkg/sessionctx/variable/sysvar.go:1418 +0x44
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:143 +0xa94
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCacheIfNeeded()
      pkg/domain/sysvar_cache.go:50 +0x1bb
  github.com/pingcap/tidb/pkg/domain.(*Domain).GetSessionCache()
      pkg/domain/sysvar_cache.go:61 +0x4a
  github.com/pingcap/tidb/pkg/session.(*session).loadCommonGlobalVariablesIfNeeded()
      pkg/session/session.go:3949 +0x2bc
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2014 +0x17a
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteInternal()
      pkg/session/session.go:1525 +0x3af
  github.com/pingcap/tidb/pkg/domain.(*Domain).LoadPrivilegeLoop()
      pkg/domain/domain.go:1901 +0x102
  github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl()
      pkg/session/session.go:3531 +0x807
  github.com/pingcap/tidb/pkg/session.BootstrapSession()
      pkg/session/session.go:3407 +0x293
  github.com/pingcap/tidb/tests/realtikvtest.CreateMockStoreAndDomainAndSetup()
      tests/realtikvtest/testkit.go:136 +0x269
  github.com/pingcap/tidb/tests/realtikvtest.CreateMockStoreAndSetup()
      tests/realtikvtest/testkit.go:109 +0xa4
  tests/realtikvtest/brietest/brietest_test.initTestKit()
      tests/realtikvtest/brietest/backup_restore_test.go:35 +0x90
  tests/realtikvtest/brietest/brietest_test.NewLogBackupKit()
      tests/realtikvtest/brietest/pitr_test.go:136 +0x3b
  tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore()
      tests/realtikvtest/brietest/pitr_test.go:573 +0x3e
  testing.tRunner()
      GOROOT/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1743 +0x44

Previous write at 0x00c056d155b0 by goroutine 243602:
  github.com/pingcap/tidb/pkg/domain.(*schemaValidator).Update()
      pkg/domain/schema_validator.go:147 +0x24e
  github.com/pingcap/tidb/pkg/domain.(*Domain).Reload()
      pkg/domain/domain.go:853 +0xe73
  github.com/pingcap/tidb/pkg/domain.(*Domain).loadSchemaInLoop()
      pkg/domain/domain.go:1149 +0x324
  github.com/pingcap/tidb/pkg/domain.(*Domain).Start.func2()
      pkg/domain/domain.go:1534 +0x52
  github.com/pingcap/tidb/pkg/util.(*WaitGroupEnhancedWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:103 +0xba

and

        	Error Trace:	pkg/testkit/testkit.go:241
        	            				pkg/testkit/testkit.go:176
        	            				tests/realtikvtest/brietest/pitr_test.go:608
        	Error:      	Received unexpected error:
        	            	[schema:1146]Table 'test.TestPiTRAndFailureRestore' doesn't exist
        	            	github.com/pingcap/errors.AddStack
        	            		external/com_github_pingcap_errors/errors.go:178
        	            	github.com/pingcap/errors.Trace
        	            		external/com_github_pingcap_errors/juju_adaptor.go:15
        	            	github.com/pingcap/tidb/pkg/planner/core.Preprocess
        	            		pkg/planner/core/preprocess.go:152
        	            	github.com/pingcap/tidb/pkg/executor.(*Compiler).Compile
        	            		pkg/executor/compiler.go:69
        	            	github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt
        	            		pkg/session/session.go:2100
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext
        	            		pkg/testkit/testkit.go:425
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQueryWithContext
        	            		pkg/testkit/testkit.go:240
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQuery
        	            		pkg/testkit/testkit.go:176
        	            	tests/realtikvtest/brietest/brietest_test.TestPiTRAndFailureRestore
        	            		tests/realtikvtest/brietest/pitr_test.go:608
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1690
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1700
        	Test:       	TestPiTRAndFailureRestore
        	Messages:   	sql:SELECT COUNT(*) FROM test.TestPiTRAndFailureRestore, args:[]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

Metadata

Metadata

Assignees

No one assigned

    Labels

    component/brThis issue is related to BR of TiDB.severity/moderatetype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions