-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Description
Under certain conditions a CEP can be kept for unmanaged pods. This can happen, for example, in case Cilium stops running and the infra container of a pod is restarted on a node where another CNI besides Cilium starts managing it.
The GC of the CEP will not happen because a) the ownerReference
of the CEP is the same since the pod is the same and b) Cilium Operator is only GC CEPs if the pod does not exist, which is also not the case. Both a) and b) are not sufficient as the only field that changed from the pod status was the pod IP.
There are two potential fixes for this
-
Watch for CEPs, which we already do under [1] and mark for deletion CEPs that are not running on the cilium agent. We can't delete them immediately since during the restore process the local agent will only start restoring the local endpoints after all state is synchronized with Kubernetes [2]. Thus, the CEPs can only be deleted after the endpoint restoration.
-
Verify if the IP addresses of the pod and the CEP are the same, if they are not, mark the CEP for deletion. This process can be executed from the cilium-operator. Similarly to case 1., we shouldn't delete the CEPs immediately at the risk of an endpoint having its IP address changed by a restart of the infra container where Cilium would be the CNI managing the pod. Cilium would likely re-create the CEP with the same name and the most up to date IP address but the IP address of the Pod Status might take a couple more seconds to be updated. If we delay the CEP deletion for maybe 1 minute we could guarantee the IP address of the Pod would be up to date and the CEP deletion would not happen unnecessarily.
[1]
cilium/pkg/k8s/watchers/cilium_endpoint.go
Lines 39 to 47 in 63c0b29
AddFunc: func(obj interface{}) { | |
var valid, equal bool | |
defer func() { k.K8sEventReceived(metricCiliumEndpoint, metricCreate, valid, equal) }() | |
if ciliumEndpoint, ok := obj.(*types.CiliumEndpoint); ok { | |
valid = true | |
k.endpointUpdated(nil, ciliumEndpoint) | |
k.K8sEventProcessed(metricCiliumEndpoint, metricCreate, true) | |
} | |
}, |
[2]
cilium/daemon/cmd/daemon_main.go
Lines 1621 to 1627 in 54fdf6e
if k8s.IsEnabled() { | |
// Wait only for certain caches, but not all! | |
// (Check Daemon.InitK8sSubsystem() for more info) | |
<-d.k8sCachesSynced | |
} | |
bootstrapStats.k8sInit.End(true) | |
restoreComplete := d.initRestore(restoredEndpoints) |