Skip to content

Warnings on endpoint deletion: unable to delete element Endpoint: X from map cilium_xxx: delete: key does not exist #35803

New issue
New issue
Open
Open
Warnings on endpoint deletion: unable to delete element Endpoint: X from map cilium_xxx: delete: key does not exist#35803
Labels
area/agentCilium agent related.kind/bugThis is a bug in the Cilium logic.kind/tech-debtTechnical debtpinnedThese issues are not marked stale by our issue bot.
@pchaigno

Description

@pchaigno
pchaigno
opened on Nov 6, 2024

Full endpoint life in agent logs (see the end for the warnings):

logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.570865730Z time="2024-11-05T12:45:29Z" level=info msg="New endpoint" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.570869467Z time="2024-11-05T12:45:29Z" level=debug msg="Skipping CiliumEndpoint update because it has no k8s cep name" ciliumEndpointName=/ containerID= containerInterface= controller="sync-to-k8s-ciliumendpoint (2373)" datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpointsynchronizer
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.591871630Z time="2024-11-05T12:45:29Z" level=debug msg="Refreshing labels of endpoint" containerID= endpointID=2373 identityLabels="reserved:health" infoLabels= sourceFilter=any subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.593117215Z time="2024-11-05T12:45:29Z" level=debug msg="Assigning security relevant label" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ obj="{Key:health Value: Source:reserved cidr:<nil>}" subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.594158768Z time="2024-11-05T12:45:29Z" level=info msg="Resolving identity labels (blocking)" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identityLabels="reserved:health" ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.594761492Z time="2024-11-05T12:45:29Z" level=debug msg="Resolving identity for labels" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identityLabels="reserved:health" ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.596558315Z time="2024-11-05T12:45:29Z" level=debug msg="Assigned new identity to endpoint" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identity=4 identityLabels="reserved:health" ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.597989124Z time="2024-11-05T12:45:29Z" level=debug msg="Set identity for this endpoint" ciliumEndpointName=/ code=OK containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 endpointState=ready ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ policyRevision=0 subsys=endpoint type=0
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.598514935Z time="2024-11-05T12:45:29Z" level=info msg="Identity of endpoint changed" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identity=4 identityLabels="reserved:health" ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ oldIdentity="no identity" subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.599191177Z time="2024-11-05T12:45:29Z" level=debug msg="Triggering endpoint regeneration due to updated security labels" ciliumEndpointName=/ code=OK containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 endpointState=waiting-to-regenerate identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ policyRevision=0 subsys=endpoint type=0
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:45:29.637144326Z time="2024-11-05T12:45:29Z" level=debug msg="Endpoint labels unchanged, skipping resolution of identity" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identity=4 identityLabels="reserved:health" ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.662469377Z time="2024-11-05T12:48:32Z" level=debug msg="exiting retrying regeneration goroutine due to endpoint being deleted" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.665019164Z time="2024-11-05T12:48:32Z" level=debug msg="Deleting endpoint" ciliumEndpointName=/ code=OK containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 endpointState=disconnecting identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ policyRevision=0 subsys=endpoint type=0
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.696938236Z time="2024-11-05T12:48:32Z" level=debug msg="removing directory" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 directory=2373 endpointID=2373 identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.701127191Z time="2024-11-05T12:48:32Z" level=debug msg="removing directory" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 directory=2373_next_fail endpointID=2373 identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.702524278Z time="2024-11-05T12:48:32Z" level=debug msg="removing directory" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 directory=2373_next endpointID=2373 identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.711643522Z time="2024-11-05T12:48:32Z" level=debug msg="Endpoint removed" ciliumEndpointName=/ code=OK containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 endpointState=disconnected identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ policyRevision=0 subsys=endpoint type=0
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.711646558Z time="2024-11-05T12:48:32Z" level=info msg="Removed endpoint" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.711649333Z time="2024-11-05T12:48:32Z" level=debug msg="Waiting for proxy updates to complete..." ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.711728270Z time="2024-11-05T12:48:32Z" level=debug msg="Wait time for proxy updates: 16.611µs" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2373 identity=4 ipv4=10.244.0.200 ipv6="fd00:10:244::6c69" k8sPodName=/ subsys=endpoint
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.711730946Z time="2024-11-05T12:48:32Z" level=warning msg="Ignoring error while deleting endpoint" endpointID=2373 error="Unable to delete key fd00:10:244::6c69:0 from /sys/fs/bpf/tc/globals/cilium_lxc: unable to delete element fd00:10:244::6c69:0 from map cilium_lxc: delete: key does not exist" subsys=daemon
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.711733310Z time="2024-11-05T12:48:32Z" level=warning msg="Ignoring error while deleting endpoint" endpointID=2373 error="Unable to delete key 10.244.0.200:0 from /sys/fs/bpf/tc/globals/cilium_lxc: unable to delete element 10.244.0.200:0 from map cilium_lxc: delete: key does not exist" subsys=daemon
logs-cilium-vf4fr-cilium-agent-20241105-125728.log:2024-11-05T12:48:32.711735784Z time="2024-11-05T12:48:32Z" level=warning msg="Ignoring error while deleting endpoint" endpointID=2373 error="removing endpoint program from global policy map: unable to delete element Endpoint: 2373 from map cilium_call_policy: delete: key does not exist" subsys=daemon

There's probably some path that can lead to a double delete of the endpoint map entries. Could be related to the fact a regeneration goroutine was interrupted by the endpoint deletion.

Sysdump: cilium-sysdump-double-ep-delete.zip

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/agentCilium agent related.kind/bugThis is a bug in the Cilium logic.kind/tech-debtTechnical debtpinnedThese issues are not marked stale by our issue bot.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions