-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Closed
Closed
Copy link
Labels
kind/bugThis is a bug in the Cilium logic.This is a bug in the Cilium logic.kind/regressionThis functionality worked fine before, but was broken in a newer release of Cilium.This functionality worked fine before, but was broken in a newer release of Cilium.sig/policyImpacts whether traffic is allowed or denied based on user-defined policies.Impacts whether traffic is allowed or denied based on user-defined policies.
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
One cilium agent pod crashed after importing a CiliumNetworkPolicy
with error fatal error: concurrent map read and map write
. I've hit this during a connectivity test here.
Cilium Version
Current latest on master: 918b75c
Kernel Version
Not relevant
Kubernetes Version
Not relevant
Sysdump
cilium-sysdump-20230224-164728.zip
Relevant log output
2023-02-24T16:47:17.128413432Z level=info msg="Imported CiliumNetworkPolicy" ciliumNetworkPolicyName=client-egress-to-cidr-deny k8sApiVersion= k8sNamespace=cilium-test subsys=k8s-watcher
2023-02-24T16:47:17.132382045Z fatal error: concurrent map read and map write
2023-02-24T16:47:17.145717662Z
2023-02-24T16:47:17.146030778Z goroutine 1620 [running]:
2023-02-24T16:47:17.146541906Z github.com/cilium/cilium/pkg/policy.(*SelectorCache).GetLabels(0xc003fec1f5?, 0xa?)
2023-02-24T16:47:17.146931027Z /go/src/github.com/cilium/cilium/pkg/policy/selectorcache.go:1094 +0x3f
2023-02-24T16:47:17.147768072Z github.com/cilium/cilium/pkg/policy.(*MapStateEntry).getNets(0xc003833f98, {0x3769b20?, 0xc0003c25b0?}, 0x0?)
2023-02-24T16:47:17.148152992Z /go/src/github.com/cilium/cilium/pkg/policy/mapstate.go:204 +0x75
2023-02-24T16:47:17.153780895Z github.com/cilium/cilium/pkg/policy.entryIdentityIsSupersetOf({0x1?, 0x0?, 0x0?, 0x0?}, {0x0, 0x1, 0x0, {0xc00223a540, 0x1, 0x1}, ...}, ...)
2023-02-24T16:47:17.154156315Z /go/src/github.com/cilium/cilium/pkg/policy/mapstate.go:381 +0x87
2023-02-24T16:47:17.154403328Z github.com/cilium/cilium/pkg/policy.MapState.denyPreferredInsertWithChanges(0x8?, {0x100003e, 0x0, 0x0, 0x1}, {0x0, 0x1, 0x0, {0xc00223a540, 0x1, ...}, ...}, ...)
2023-02-24T16:47:17.154410129Z /go/src/github.com/cilium/cilium/pkg/policy/mapstate.go:435 +0xb25
2023-02-24T16:47:17.154413529Z github.com/cilium/cilium/pkg/policy.MapState.DenyPreferredInsert(...)
2023-02-24T16:47:17.154417129Z /go/src/github.com/cilium/cilium/pkg/policy/mapstate.go:285
2023-02-24T16:47:17.154420529Z github.com/cilium/cilium/pkg/policy.(*EndpointPolicy).computeDirectionL4PolicyMapEntries(0xc0017745c0, 0xc0017745c0?, 0xc003f4f51c?, 0xc0?)
2023-02-24T16:47:17.154423629Z /go/src/github.com/cilium/cilium/pkg/policy/resolve.go:191 +0x2c5
2023-02-24T16:47:17.159766616Z github.com/cilium/cilium/pkg/policy.(*EndpointPolicy).computeDesiredL4PolicyMapEntries(0xc0017745c0)
2023-02-24T16:47:17.159780717Z /go/src/github.com/cilium/cilium/pkg/policy/resolve.go:164 +0x56
2023-02-24T16:47:17.159793418Z github.com/cilium/cilium/pkg/policy.(*selectorPolicy).DistillPolicy(0xc0003cae60, {0x37a5b00?, 0xc0003df180}, 0x0)
2023-02-24T16:47:17.159796818Z /go/src/github.com/cilium/cilium/pkg/policy/resolve.go:141 +0x105
2023-02-24T16:47:17.159799818Z github.com/cilium/cilium/pkg/policy.(*cachedSelectorPolicy).Consume(0xc000602a80?, {0x37a5b00?, 0xc0003df180?})
2023-02-24T16:47:17.159802918Z /go/src/github.com/cilium/cilium/pkg/policy/distillery.go:202 +0x32
2023-02-24T16:47:17.159805918Z github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regeneratePolicy(0xc0003df180)
2023-02-24T16:47:17.159808919Z /go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:237 +0x40e
2023-02-24T16:47:17.159811919Z github.com/cilium/cilium/pkg/endpoint.(*Endpoint).runPreCompilationSteps(0xc0003df180, 0xc002df2c00)
2023-02-24T16:47:17.159815719Z /go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:844 +0x2c7
2023-02-24T16:47:17.159819219Z github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regenerateBPF(0xc0003df180, 0xc002df2c00)
2023-02-24T16:47:17.159822319Z /go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:603 +0x189
2023-02-24T16:47:17.159825319Z github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regenerate(0xc0003df180, 0xc002df2c00)
2023-02-24T16:47:17.159828320Z /go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:401 +0x7a5
2023-02-24T16:47:17.159831520Z github.com/cilium/cilium/pkg/endpoint.(*EndpointRegenerationEvent).Handle(0xc003768830, 0x2ad97c0?)
2023-02-24T16:47:17.159834620Z /go/src/github.com/cilium/cilium/pkg/endpoint/events.go:53 +0x325
2023-02-24T16:47:17.159837720Z github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run.func1()
2023-02-24T16:47:17.159840620Z /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:245 +0x142
2023-02-24T16:47:17.159844220Z sync.(*Once).doSlow(0xc001c29e01?, 0x4430e5?)
2023-02-24T16:47:17.159847521Z /usr/local/go/src/sync/once.go:74 +0xc2
2023-02-24T16:47:17.159851021Z sync.(*Once).Do(...)
2023-02-24T16:47:17.159854021Z /usr/local/go/src/sync/once.go:65
2023-02-24T16:47:17.159857121Z github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run(0xc000e45c80?)
2023-02-24T16:47:17.159860521Z /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:233 +0x45
2023-02-24T16:47:17.159863521Z created by github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).Run
2023-02-24T16:47:17.159866422Z /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:229 +0x76
Anything else?
This function seems to be the culprit based on the stack trace above:
cilium/pkg/policy/selectorcache.go
Lines 1093 to 1099 in 918b75c
func (sc *SelectorCache) GetLabels(id identity.NumericIdentity) labels.LabelArray { | |
ident, ok := sc.idCache[id] | |
if !ok { | |
return labels.LabelArray{} | |
} | |
return ident.lbls | |
} |
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
kind/bugThis is a bug in the Cilium logic.This is a bug in the Cilium logic.kind/regressionThis functionality worked fine before, but was broken in a newer release of Cilium.This functionality worked fine before, but was broken in a newer release of Cilium.sig/policyImpacts whether traffic is allowed or denied based on user-defined policies.Impacts whether traffic is allowed or denied based on user-defined policies.