Skip to content

[DATA RACE]: github.com/cilium/cilium/pkg/endpoint.(*Endpoint).toSerializedEndpoint() #38324

New issue
New issue
Closed
#39429
Closed
[DATA RACE]: github.com/cilium/cilium/pkg/endpoint.(*Endpoint).toSerializedEndpoint()#38324
#39429
Assignees
ti-mo
Labels
area/agentCilium agent related.kind/bugThis is a bug in the Cilium logic.kind/bug/race-detectorThis is a bug identified by concurrency tests.release-blocker/1.18This issue will prevent the release of the next version of Cilium.
@aanm

Description

@aanm
aanm
opened on Mar 19, 2025

The following race detector was found:

WARNING: DATA RACE
Read at 0x00c00055bfa8 by goroutine 23384:
  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).toSerializedEndpoint()
      /go/src/github.com/cilium/cilium/pkg/endpoint/restore.go:439 +0xf9e
  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).MarshalJSON()
      /go/src/github.com/cilium/cilium/pkg/endpoint/restore.go:583 +0x26
  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).writeHeaderfile()
      /go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:175 +0x2c7
  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regenerateBPF()
      /go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:432 +0x924
  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regenerate()
      /go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:477 +0xe5c
  github.com/cilium/cilium/pkg/endpoint.(*EndpointRegenerationEvent).Handle()
      /go/src/github.com/cilium/cilium/pkg/endpoint/events.go:77 +0x67c
  github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).Run.gowrap1.(*EventQueue).run.1()
      /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:244 +0x1f4
  sync.(*Once).doSlow()
      /usr/local/go/src/sync/once.go:78 +0xe1
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:69 +0x44
  github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run()
      /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:232 +0x69
  github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).Run.gowrap1()
      /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:228 +0x17

Previous write at 0x00c00055bfa8 by goroutine 23388:
  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).SetCiliumEndpointUID()
      /go/src/github.com/cilium/cilium/pkg/endpoint/identifiers.go:161 +0x56
  github.com/cilium/cilium/pkg/endpointmanager.(*EndpointSynchronizer).RunK8sCiliumEndpointSync.func1()
      /go/src/github.com/cilium/cilium/pkg/endpointmanager/endpointsynchronizer.go:204 +0x124c
  github.com/cilium/cilium/pkg/controller.(*controller).runController()
      /go/src/github.com/cilium/cilium/pkg/controller/controller.go:251 +0x9c
  github.com/cilium/cilium/pkg/controller.(*Manager).createControllerLocked.gowrap1()
      /go/src/github.com/cilium/cilium/pkg/controller/manager.go:116 +0xbc

Goroutine 23384 (running) created at:
  github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).Run()
      /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:228 +0xf5
  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).Start()
      /go/src/github.com/cilium/cilium/pkg/endpoint/events.go:359 +0x291
  github.com/cilium/cilium/pkg/endpointmanager.(*endpointManager).expose()
      /go/src/github.com/cilium/cilium/pkg/endpointmanager/manager.go:657 +0x14d
  github.com/cilium/cilium/pkg/endpointmanager.(*endpointManager).AddEndpoint()
      /go/src/github.com/cilium/cilium/pkg/endpointmanager/manager.go:709 +0x7a7
  github.com/cilium/cilium/daemon/cmd.(*Daemon).createEndpoint()
      /go/src/github.com/cilium/cilium/daemon/cmd/endpoint.go:538 +0x2781
  github.com/cilium/cilium/daemon/cmd.putEndpointIDHandler()
      /go/src/github.com/cilium/cilium/daemon/cmd/endpoint.go:666 +0x6d3
  github.com/cilium/cilium/daemon/cmd.(*apiHandler[go.shape.struct { HTTPRequest *net/http.Request "json:\"-\""; Endpoint *github.com/cilium/cilium/api/v1/models.EndpointChangeRequest; ID string }]).Handle()
      /go/src/github.com/cilium/cilium/daemon/cmd/api_handlers.go:64 +0xe8
  github.com/cilium/cilium/daemon/cmd.(*apiHandler[github.com/cilium/cilium/api/v1/server/restapi/endpoint.PutEndpointIDParams]).Handle()
      /go/src/github.com/cilium/cilium/daemon/cmd/api_handlers.go:55 +0x78
  github.com/cilium/cilium/api/v1/server/restapi/endpoint.(*PutEndpointID).ServeHTTP()
      /go/src/github.com/cilium/cilium/api/v1/server/restapi/endpoint/put_endpoint_id.go:58 +0x2d3
  github.com/go-openapi/runtime/middleware.(*Context).RoutesHandler.NewOperationExecutor.func1()
      /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/operation.go:28 +0x8a
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2294 +0x47
  github.com/go-openapi/runtime/middleware.NewRouter.func1()
      /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/router.go:80 +0x2fc
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2294 +0x47
  github.com/go-openapi/runtime/middleware.Redoc.serveUI.func1()
      /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/ui_options.go:164 +0x10c
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2294 +0x47
  github.com/go-openapi/runtime/middleware.Spec.func1()
      /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/spec.go:72 +0x224
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2294 +0x47
  github.com/cilium/cilium/pkg/metrics.(*APIEventTSHelper).ServeHTTP()
      /go/src/github.com/cilium/cilium/pkg/metrics/middleware.go:63 +0x3b0
  github.com/cilium/cilium/pkg/api.(*APIPanicHandler).ServeHTTP()
      /go/src/github.com/cilium/cilium/pkg/api/apipanic.go:56 +0x115
  net/http.(*ServeMux).ServeHTTP()
      /usr/local/go/src/net/http/server.go:2822 +0x23e
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:3301 +0x2a1
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:2102 +0x1304
  net/http.(*Server).Serve.gowrap3()
      /usr/local/go/src/net/http/server.go:3454 +0x4f

Goroutine 23388 (running) created at:
  github.com/cilium/cilium/pkg/controller.(*Manager).createControllerLocked()
      /go/src/github.com/cilium/cilium/pkg/controller/manager.go:116 +0x757
  github.com/cilium/cilium/pkg/controller.(*Manager).updateController()
      /go/src/github.com/cilium/cilium/pkg/controller/manager.go:89 +0x5b8
  github.com/cilium/cilium/pkg/controller.(*Manager).UpdateController()
      /go/src/github.com/cilium/cilium/pkg/controller/manager.go:53 +0xbe7
  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).UpdateController()
      /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:495 +0xb3f
  github.com/cilium/cilium/pkg/endpointmanager.(*EndpointSynchronizer).RunK8sCiliumEndpointSync()
      /go/src/github.com/cilium/cilium/pkg/endpointmanager/endpointsynchronizer.go:101 +0xb4e
  github.com/cilium/cilium/pkg/endpointmanager.(*endpointManager).expose()
      /go/src/github.com/cilium/cilium/pkg/endpointmanager/manager.go:664 +0x481
  github.com/cilium/cilium/pkg/endpointmanager.(*endpointManager).AddEndpoint()
      /go/src/github.com/cilium/cilium/pkg/endpointmanager/manager.go:709 +0x7a7
  github.com/cilium/cilium/daemon/cmd.(*Daemon).createEndpoint()
      /go/src/github.com/cilium/cilium/daemon/cmd/endpoint.go:538 +0x2781
  github.com/cilium/cilium/daemon/cmd.putEndpointIDHandler()
      /go/src/github.com/cilium/cilium/daemon/cmd/endpoint.go:666 +0x6d3
  github.com/cilium/cilium/daemon/cmd.(*apiHandler[go.shape.struct { HTTPRequest *net/http.Request "json:\"-\""; Endpoint *github.com/cilium/cilium/api/v1/models.EndpointChangeRequest; ID string }]).Handle()
      /go/src/github.com/cilium/cilium/daemon/cmd/api_handlers.go:64 +0xe8
  github.com/cilium/cilium/daemon/cmd.(*apiHandler[github.com/cilium/cilium/api/v1/server/restapi/endpoint.PutEndpointIDParams]).Handle()
      /go/src/github.com/cilium/cilium/daemon/cmd/api_handlers.go:55 +0x78
  github.com/cilium/cilium/api/v1/server/restapi/endpoint.(*PutEndpointID).ServeHTTP()
      /go/src/github.com/cilium/cilium/api/v1/server/restapi/endpoint/put_endpoint_id.go:58 +0x2d3
  github.com/go-openapi/runtime/middleware.(*Context).RoutesHandler.NewOperationExecutor.func1()
      /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/operation.go:28 +0x8a
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2294 +0x47
  github.com/go-openapi/runtime/middleware.NewRouter.func1()
      /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/router.go:80 +0x2fc
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2294 +0x47
  github.com/go-openapi/runtime/middleware.Redoc.serveUI.func1()
      /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/ui_options.go:164 +0x10c
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2294 +0x47
  github.com/go-openapi/runtime/middleware.Spec.func1()
      /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/spec.go:72 +0x224
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2294 +0x47
  github.com/cilium/cilium/pkg/metrics.(*APIEventTSHelper).ServeHTTP()
      /go/src/github.com/cilium/cilium/pkg/metrics/middleware.go:63 +0x3b0
  github.com/cilium/cilium/pkg/api.(*APIPanicHandler).ServeHTTP()
      /go/src/github.com/cilium/cilium/pkg/api/apipanic.go:56 +0x115
  net/http.(*ServeMux).ServeHTTP()
      /usr/local/go/src/net/http/server.go:2822 +0x23e
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:3301 +0x2a1
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:2102 +0x1304
  net/http.(*Server).Serve.gowrap3()
      /usr/local/go/src/net/http/server.go:3454 +0x4f

This was likely caused by af09899 which moved this section

cilium/pkg/endpoint/bpf.go

Lines 430 to 442 in 5a54846

dir := datapathRegenCtxt.currentDir
if datapathRegenCtxt.regenerationLevel >= regeneration.RegenerateWithDatapath {
if err := e.writeHeaderfile(datapathRegenCtxt.nextDir); err != nil {
return 0, fmt.Errorf("write endpoint header file: %w", err)
}
dir = datapathRegenCtxt.nextDir
}
if err := e.lockAlive(); err != nil {
return 0, err
}
datapathRegenCtxt.epInfoCache = e.createEpInfoCache(dir)
e.unlock()
to be outside of lock mutex. The function documentation does mention that the mutex must be held before calling writeHeaderfile:

cilium/pkg/endpoint/bpf.go

Lines 165 to 168 in 5a54846

// writeHeaderfile writes the lxc_config.h header file of an endpoint.
//
// e.mutex must be write-locked.
func (e *Endpoint) writeHeaderfile(prefix string) error {

cc @ti-mo

Metadata

Metadata

Assignees

Labels

area/agentCilium agent related.kind/bugThis is a bug in the Cilium logic.kind/bug/race-detectorThis is a bug identified by concurrency tests.release-blocker/1.18This issue will prevent the release of the next version of Cilium.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions