cilium endpoint regenerate broken #12005
Description
root@apoc:~/go/src/github.com/cilium/cilium# ./cilium/cilium endpoint list
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
84 Disabled Disabled 7008 no labels 2001:db8:5::4817 10.29.71.199 ready
1038 Disabled Disabled 7008 no labels 2001:db8:5::60a9 10.29.218.124 ready
1184 Disabled Disabled 4 reserved:health 2001:db8:5::f395 10.29.52.34 ready
2817 Disabled Disabled 1 reserved:host ready
root@apoc:~/go/src/github.com/cilium/cilium# ./cilium/cilium endpoint regenerate 84
Endpoint 84 successfully regenerated
Agent log, looks like it's trying to generate a per-EP CT map:
level=error msg="Failed to compile bpf_lxc.o: exit status 1" compiler-pid=20856 linker-pid=20857 subsys=datapath-loader
level=warning msg="In file included from /var/lib/cilium/bpf/bpf_lxc.c:8:" subsys=datapath-loader
level=warning msg="/var/run/cilium/state/globals/node_config.h:22:9: error: 'CT_MAP_TCP4' macro redefined [-Werror,-Wmacro-redefined]" subsys=datapath-loader
level=warning msg="#define CT_MAP_TCP4 cilium_ct4_global" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/ep_config.h:39:9: note: previous definition is here" subsys=datapath-loader
level=warning msg="#define CT_MAP_TCP4 test_cilium_ct_tcp4_65535" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="In file included from /var/lib/cilium/bpf/bpf_lxc.c:8:" subsys=datapath-loader
level=warning msg="/var/run/cilium/state/globals/node_config.h:23:9: error: 'CT_MAP_ANY4' macro redefined [-Werror,-Wmacro-redefined]" subsys=datapath-loader
level=warning msg="#define CT_MAP_ANY4 cilium_ct_any4_global" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/ep_config.h:40:9: note: previous definition is here" subsys=datapath-loader
level=warning msg="#define CT_MAP_ANY4 test_cilium_ct_any4_65535" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="In file included from /var/lib/cilium/bpf/bpf_lxc.c:8:" subsys=datapath-loader
level=warning msg="/var/run/cilium/state/globals/node_config.h:24:9: error: 'CT_MAP_TCP6' macro redefined [-Werror,-Wmacro-redefined]" subsys=datapath-loader
level=warning msg="#define CT_MAP_TCP6 cilium_ct6_global" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/ep_config.h:37:9: note: previous definition is here" subsys=datapath-loader
level=warning msg="#define CT_MAP_TCP6 test_cilium_ct_tcp6_65535" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="In file included from /var/lib/cilium/bpf/bpf_lxc.c:8:" subsys=datapath-loader
level=warning msg="/var/run/cilium/state/globals/node_config.h:25:9: error: 'CT_MAP_ANY6' macro redefined [-Werror,-Wmacro-redefined]" subsys=datapath-loader
level=warning msg="#define CT_MAP_ANY6 cilium_ct_any6_global" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/ep_config.h:38:9: note: previous definition is here" subsys=datapath-loader
level=warning msg="#define CT_MAP_ANY6 test_cilium_ct_any6_65535" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="In file included from /var/lib/cilium/bpf/bpf_lxc.c:8:" subsys=datapath-loader
level=warning msg="/var/run/cilium/state/globals/node_config.h:26:9: error: 'CT_MAP_SIZE_TCP' macro redefined [-Werror,-Wmacro-redefined]" subsys=datapath-loader
level=warning msg="#define CT_MAP_SIZE_TCP 524288" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/ep_config.h:41:9: note: previous definition is here" subsys=datapath-loader
level=warning msg="#define CT_MAP_SIZE_TCP 4096" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="In file included from /var/lib/cilium/bpf/bpf_lxc.c:8:" subsys=datapath-loader
level=warning msg="/var/run/cilium/state/globals/node_config.h:27:9: error: 'CT_MAP_SIZE_ANY' macro redefined [-Werror,-Wmacro-redefined]" subsys=datapath-loader
level=warning msg="#define CT_MAP_SIZE_ANY 262144" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/ep_config.h:42:9: note: previous definition is here" subsys=datapath-loader
level=warning msg="#define CT_MAP_SIZE_ANY 4096" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="In file included from /var/lib/cilium/bpf/bpf_lxc.c:22:" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/lib/icmp6.h:33:29: error: use of undeclared identifier 'NODE_MAC'" subsys=datapath-loader
level=warning msg=" union macaddr smac, dmac = NODE_MAC;" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/lib/icmp6.h:342:30: error: use of undeclared identifier 'NODE_MAC'" subsys=datapath-loader
level=warning msg=" union macaddr router_mac = NODE_MAC;" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="In file included from /var/lib/cilium/bpf/bpf_lxc.c:26:" subsys=datapath-loader
level=warning msg="/var/lib/cilium/bpf/lib/lxc.h:67:29: error: use of undeclared identifier 'NODE_MAC'" subsys=datapath-loader
level=warning msg=" union macaddr router_mac = NODE_MAC;" subsys=datapath-loader
level=warning msg=" ^" subsys=datapath-loader
level=warning msg="9 errors generated." subsys=datapath-loader
level=warning msg="JoinEP: Failed to compile" containerID=f93fd3aa58 datapathPolicyRevision=1 debug=false desiredPolicyRevision=1 endpointID=84 error="Failed to compile bpf_lxc.o: exit status 1" identity=7008 ipv4=10.29.71.199 ipv6="2001:db8:5::4817" k8sPodName=/ params="&{Source:bpf_lxc.c Output:bpf_lxc.o OutputType:obj}" subsys=datapath-loader
level=info msg="Regenerated endpoint BPF program" containerID=f93fd3aa58 datapathPolicyRevision=1 desiredPolicyRevision=1 endpointID=84 error="Failed to compile bpf_lxc.o: exit status 1" identity=7008 ipv4=10.29.71.199 ipv6="2001:db8:5::4817" k8sPodName=/ subsys=endpoint
level=warning msg="generating BPF for endpoint failed, keeping stale directory." containerID=f93fd3aa58 datapathPolicyRevision=1 desiredPolicyRevision=1 endpointID=84 file-path=84_next_fail identity=7008 ipv4=10.29.71.199 ipv6="2001:db8:5::4817" k8sPodName=/ subsys=endpoint
level=warning msg="Regeneration of endpoint failed" bpfCompilation=92.742911ms bpfLoadProg=0s bpfWaitForELF=0s bpfWriteELF=0s buildDuration=93.000511ms containerID=f93fd3aa58 datapathPolicyRevision=1 desiredPolicyRevision=1 endpointID=84 error="Failed to compile bpf_lxc.o: exit status 1" identity=7008 ipv4=10.29.71.199 ipv6="2001:db8:5::4817" k8sPodName=/ mapSync=806ns policyCalculation="2.993µs" prepareBuild="72.888µs" proxyConfiguration="5.486µs" proxyPolicyCalculation="81.856µs" proxyWaitForAck=0s reason="endpoint was manually regenerated via API" subsys=endpoint waitingForCTClean=203ns waitingForLock="11.691µs"
level=error msg="endpoint regeneration failed" containerID=f93fd3aa58 datapathPolicyRevision=1 desiredPolicyRevision=1 endpointID=84 error="Failed to compile bpf_lxc.o: exit status 1" identity=7008 ipv4=10.29.71.199 ipv6="2001:db8:5::4817" k8sPodName=/ subsys=endpoint
level=info msg="Rewrote endpoint BPF program" containerID=f93fd3aa58 datapathPolicyRevision=1 desiredPolicyRevision=1 endpointID=84 identity=7008 ipv4=10.29.71.199 ipv6="2001:db8:5::4817" k8sPodName=/ subsys=endpoint
Despite ConntrackLocal being disabled:
# ./cilium/cilium config
Conntrack Enabled
ConntrackAccounting Enabled
ConntrackLocal Disabled
Debug Disabled
DebugLB Disabled
DropNotification Enabled
MonitorAggregationLevel None
PolicyAuditMode Disabled
PolicyTracing Disabled
PolicyVerdictNotification Enabled
TraceNotification Enabled
k8s-configuration
k8s-endpoint
PolicyEnforcement default
# ./cilium/cilium endpoint config 84
Conntrack Enabled
ConntrackAccounting Enabled
ConntrackLocal Disabled
Debug Disabled
DebugLB Disabled
DropNotification Enabled
MonitorAggregationLevel None
NAT46 Disabled
PolicyAuditMode Disabled
PolicyVerdictNotification Enabled
TraceNotification Enabled