Cilium agent may fail to start due to crashing ip(6)tables command #22482
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
Cilium agent failed to start in a CI test with these logs:
2022-12-01T11:44:36.376838195Z level=warning msg="Failed to install iptables rules" error="unable to run 'ip6tables -t nat -S' iptables command: signal: aborted (core dumped) stderr=\"free(): double free detected in tcache 2\\n\"" subsys=iptables
2022-12-01T11:45:16.511342364Z level=warning msg="Failed to install iptables rules" error="unable to run 'ip6tables -t nat -S' iptables command: signal: aborted (core dumped) stderr=\"free(): double free detected in tcache 2\\n\"" subsys=iptables
2022-12-01T11:46:36.626712414Z level=error msg="Start hook failed" error="daemon creation failed: error while initializing daemon: failed while reinitializing datapath: failed to install iptables rules: unable to run 'ip6tables -t nat -S' iptables command: signal: aborted (core dumped) stderr=\"free(): double free detected in tcache 2\\n\"" function="cmd.newDaemonPromise.func1 (daemon_main.go:1660)" subsys=hive
2022-12-01T11:46:36.626734214Z level=info msg="Stop hook executed" duration="14.2µs" function="client.(*compositeClientset).onStop" subsys=hive
2022-12-01T11:46:36.626861513Z level=info msg="Stopped gops server" address="127.0.0.1:9890" subsys=gops
2022-12-01T11:46:36.626867313Z level=info msg="Stop hook executed" duration="117.299µs" function="gops.registerGopsHooks.func2 (cell.go:51)" subsys=hive
2022-12-01T11:46:36.626871513Z level=fatal msg="failed to start: daemon creation failed: error while initializing daemon: failed while reinitializing datapath: failed to install iptables rules: unable to run 'ip6tables -t nat -S' iptables command: signal: aborted (core dumped) stderr=\"free(): double free detected in tcache 2\\n\"" subsys=daemon
This bug has been fixed in upstream iptables: https://git.netfilter.org/iptables/commit/iptables/?id=4318961230bce82958df82b57f1796143bf2f421. First tag after the fix is 1.8.8 (tagged on 2022-05-13).
Apparently the iptables version shipped in Cilium agent container is based on 1.8.7: https://packages.ubuntu.com/jammy/iptables
Cilium Version
master at 378da16
Kernel Version
kernel 5.15.0
Kubernetes Version
N/A
Sysdump
cilium-sysdump-out.zip (18).zip
Relevant log output
No response
Anything else?
No response
Code of Conduct
- I agree to follow this project's Code of Conduct