Skip to content

concurrent map read and map write panic after importing CiliumNetworkPolicy #24021

@giorio94

Description

@giorio94

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

One cilium agent pod crashed after importing a CiliumNetworkPolicy with error fatal error: concurrent map read and map write. I've hit this during a connectivity test here.

Cilium Version

Current latest on master: 918b75c

Kernel Version

Not relevant

Kubernetes Version

Not relevant

Sysdump

cilium-sysdump-20230224-164728.zip

Relevant log output

2023-02-24T16:47:17.128413432Z level=info msg="Imported CiliumNetworkPolicy" ciliumNetworkPolicyName=client-egress-to-cidr-deny k8sApiVersion= k8sNamespace=cilium-test subsys=k8s-watcher
2023-02-24T16:47:17.132382045Z fatal error: concurrent map read and map write
2023-02-24T16:47:17.145717662Z 
2023-02-24T16:47:17.146030778Z goroutine 1620 [running]:
2023-02-24T16:47:17.146541906Z github.com/cilium/cilium/pkg/policy.(*SelectorCache).GetLabels(0xc003fec1f5?, 0xa?)
2023-02-24T16:47:17.146931027Z 	/go/src/github.com/cilium/cilium/pkg/policy/selectorcache.go:1094 +0x3f
2023-02-24T16:47:17.147768072Z github.com/cilium/cilium/pkg/policy.(*MapStateEntry).getNets(0xc003833f98, {0x3769b20?, 0xc0003c25b0?}, 0x0?)
2023-02-24T16:47:17.148152992Z 	/go/src/github.com/cilium/cilium/pkg/policy/mapstate.go:204 +0x75
2023-02-24T16:47:17.153780895Z github.com/cilium/cilium/pkg/policy.entryIdentityIsSupersetOf({0x1?, 0x0?, 0x0?, 0x0?}, {0x0, 0x1, 0x0, {0xc00223a540, 0x1, 0x1}, ...}, ...)
2023-02-24T16:47:17.154156315Z 	/go/src/github.com/cilium/cilium/pkg/policy/mapstate.go:381 +0x87
2023-02-24T16:47:17.154403328Z github.com/cilium/cilium/pkg/policy.MapState.denyPreferredInsertWithChanges(0x8?, {0x100003e, 0x0, 0x0, 0x1}, {0x0, 0x1, 0x0, {0xc00223a540, 0x1, ...}, ...}, ...)
2023-02-24T16:47:17.154410129Z 	/go/src/github.com/cilium/cilium/pkg/policy/mapstate.go:435 +0xb25
2023-02-24T16:47:17.154413529Z github.com/cilium/cilium/pkg/policy.MapState.DenyPreferredInsert(...)
2023-02-24T16:47:17.154417129Z 	/go/src/github.com/cilium/cilium/pkg/policy/mapstate.go:285
2023-02-24T16:47:17.154420529Z github.com/cilium/cilium/pkg/policy.(*EndpointPolicy).computeDirectionL4PolicyMapEntries(0xc0017745c0, 0xc0017745c0?, 0xc003f4f51c?, 0xc0?)
2023-02-24T16:47:17.154423629Z 	/go/src/github.com/cilium/cilium/pkg/policy/resolve.go:191 +0x2c5
2023-02-24T16:47:17.159766616Z github.com/cilium/cilium/pkg/policy.(*EndpointPolicy).computeDesiredL4PolicyMapEntries(0xc0017745c0)
2023-02-24T16:47:17.159780717Z 	/go/src/github.com/cilium/cilium/pkg/policy/resolve.go:164 +0x56
2023-02-24T16:47:17.159793418Z github.com/cilium/cilium/pkg/policy.(*selectorPolicy).DistillPolicy(0xc0003cae60, {0x37a5b00?, 0xc0003df180}, 0x0)
2023-02-24T16:47:17.159796818Z 	/go/src/github.com/cilium/cilium/pkg/policy/resolve.go:141 +0x105
2023-02-24T16:47:17.159799818Z github.com/cilium/cilium/pkg/policy.(*cachedSelectorPolicy).Consume(0xc000602a80?, {0x37a5b00?, 0xc0003df180?})
2023-02-24T16:47:17.159802918Z 	/go/src/github.com/cilium/cilium/pkg/policy/distillery.go:202 +0x32
2023-02-24T16:47:17.159805918Z github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regeneratePolicy(0xc0003df180)
2023-02-24T16:47:17.159808919Z 	/go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:237 +0x40e
2023-02-24T16:47:17.159811919Z github.com/cilium/cilium/pkg/endpoint.(*Endpoint).runPreCompilationSteps(0xc0003df180, 0xc002df2c00)
2023-02-24T16:47:17.159815719Z 	/go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:844 +0x2c7
2023-02-24T16:47:17.159819219Z github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regenerateBPF(0xc0003df180, 0xc002df2c00)
2023-02-24T16:47:17.159822319Z 	/go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:603 +0x189
2023-02-24T16:47:17.159825319Z github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regenerate(0xc0003df180, 0xc002df2c00)
2023-02-24T16:47:17.159828320Z 	/go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:401 +0x7a5
2023-02-24T16:47:17.159831520Z github.com/cilium/cilium/pkg/endpoint.(*EndpointRegenerationEvent).Handle(0xc003768830, 0x2ad97c0?)
2023-02-24T16:47:17.159834620Z 	/go/src/github.com/cilium/cilium/pkg/endpoint/events.go:53 +0x325
2023-02-24T16:47:17.159837720Z github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run.func1()
2023-02-24T16:47:17.159840620Z 	/go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:245 +0x142
2023-02-24T16:47:17.159844220Z sync.(*Once).doSlow(0xc001c29e01?, 0x4430e5?)
2023-02-24T16:47:17.159847521Z 	/usr/local/go/src/sync/once.go:74 +0xc2
2023-02-24T16:47:17.159851021Z sync.(*Once).Do(...)
2023-02-24T16:47:17.159854021Z 	/usr/local/go/src/sync/once.go:65
2023-02-24T16:47:17.159857121Z github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run(0xc000e45c80?)
2023-02-24T16:47:17.159860521Z 	/go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:233 +0x45
2023-02-24T16:47:17.159863521Z created by github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).Run
2023-02-24T16:47:17.159866422Z 	/go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:229 +0x76

Anything else?

This function seems to be the culprit based on the stack trace above:

func (sc *SelectorCache) GetLabels(id identity.NumericIdentity) labels.LabelArray {
ident, ok := sc.idCache[id]
if !ok {
return labels.LabelArray{}
}
return ident.lbls
}

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

Labels

kind/bugThis is a bug in the Cilium logic.kind/regressionThis functionality worked fine before, but was broken in a newer release of Cilium.sig/policyImpacts whether traffic is allowed or denied based on user-defined policies.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions