Skip to content
This repository was archived by the owner on Dec 7, 2023. It is now read-only.

Chain firewall plugin to fix routing for default CNI bridge #442

Merged
merged 1 commit into from
Sep 20, 2019
Merged

Chain firewall plugin to fix routing for default CNI bridge #442

merged 1 commit into from
Sep 20, 2019

Conversation

stealthybox
Copy link
Contributor

@stealthybox stealthybox commented Sep 19, 2019 •
edited
Loading

Alternative to #427 using the firewall CNI plugin in our conflist
Fixes #401
Fixes #418

This works on my machine with the normal quirks that we've run into switching subnets.
Please test.

Output with containerd+cni working twice and docker+cni working once:

# ~/Repos/ignite-scratch/ignite-clean.sh; ~/Repos/ignite-scratch/iptables-clean-cni-ignite.sh

# make build-all

export suffix="" # use build
export ignite="sudo $( ls ~/Repos/ignite/bin/amd64/ignite${suffix} )"
test-vm() {
  r=$1 ; n=$2 ; echo $r $n
    name=${3:-t_${r}_${n}_}
    $=ignite vm run weaveworks/ignite-ubuntu --runtime $r --network-plugin $n --ssh --name $name  &&  sleep 2; sudo iptables -L -t nat | grep ignite
    $=ignite exec $name -- ping -c2 -i.1 8.8.4.4
};
#test-vm docker docker-bridge
test-vm containerd cni ctrd${suffix}_1
test-vm containerd cni ctrd${suffix}_2
test-vm docker cni

containerd cni
INFO[0000] Created VM with ID "87126a6fa47f08f3" and name "ctrd_1" 
INFO[0001] Networking is handled by "cni"               
INFO[0001] Started Firecracker VM "87126a6fa47f08f3" in a container with ID "ignite-87126a6fa47f08f3" 
CNI-00de49e45305e5d1fd4153c6  all  --  172.18.0.234         anywhere             /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
ACCEPT     all  --  anywhere             172.18.0.0/16        /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4  /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
64 bytes from 8.8.4.4: icmp_seq=1 ttl=53 time=14.7 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=53 time=16.5 ms

--- 8.8.4.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 100ms
rtt min/avg/max/mdev = 14.705/15.613/16.522/0.917 ms
containerd cni
INFO[0000] Created VM with ID "117328942cdb4496" and name "ctrd_2" 
INFO[0001] Networking is handled by "cni"               
INFO[0001] Started Firecracker VM "117328942cdb4496" in a container with ID "ignite-117328942cdb4496" 
CNI-00de49e45305e5d1fd4153c6  all  --  172.18.0.234         anywhere             /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
CNI-412ef007d5c6d81acafb143a  all  --  172.18.0.235         anywhere             /* name: "ignite-containerd-bridge" id: "ignite-117328942cdb4496" */
ACCEPT     all  --  anywhere             172.18.0.0/16        /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4  /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
ACCEPT     all  --  anywhere             172.18.0.0/16        /* name: "ignite-containerd-bridge" id: "ignite-117328942cdb4496" */
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4  /* name: "ignite-containerd-bridge" id: "ignite-117328942cdb4496" */
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
64 bytes from 8.8.4.4: icmp_seq=1 ttl=53 time=17.8 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=53 time=11.7 ms

--- 8.8.4.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 100ms
rtt min/avg/max/mdev = 11.774/14.812/17.850/3.038 ms
docker cni
INFO[0000] Created VM with ID "5bec20b7a6927eed" and name "t_docker_cni_" 
INFO[0001] Networking is handled by "cni"               
INFO[0001] Started Firecracker VM "5bec20b7a6927eed" in a container with ID "9f7522163fd27605d8b790775584b22f826f7d3b98bcb4469a5bbf70f0c0de52" 
CNI-00de49e45305e5d1fd4153c6  all  --  172.18.0.234         anywhere             /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
CNI-412ef007d5c6d81acafb143a  all  --  172.18.0.235         anywhere             /* name: "ignite-containerd-bridge" id: "ignite-117328942cdb4496" */
CNI-0428dc85ddf50abc1e01631a  all  --  172.18.0.239         anywhere             /* name: "ignite-containerd-bridge" id: "9f7522163fd27605d8b790775584b22f826f7d3b98bcb4469a5bbf70f0c0de52" */
ACCEPT     all  --  anywhere             172.18.0.0/16        /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4  /* name: "ignite-containerd-bridge" id: "ignite-87126a6fa47f08f3" */
ACCEPT     all  --  anywhere             172.18.0.0/16        /* name: "ignite-containerd-bridge" id: "9f7522163fd27605d8b790775584b22f826f7d3b98bcb4469a5bbf70f0c0de52" */
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4  /* name: "ignite-containerd-bridge" id: "9f7522163fd27605d8b790775584b22f826f7d3b98bcb4469a5bbf70f0c0de52" */
ACCEPT     all  --  anywhere             172.18.0.0/16        /* name: "ignite-containerd-bridge" id: "ignite-117328942cdb4496" */
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4  /* name: "ignite-containerd-bridge" id: "ignite-117328942cdb4496" */
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
64 bytes from 8.8.4.4: icmp_seq=1 ttl=53 time=16.2 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=53 time=13.7 ms

--- 8.8.4.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 101ms
rtt min/avg/max/mdev = 13.793/15.024/16.255/1.231 ms


# notice there are FORWARD rules per vm
sudo iptables -S | grep CNI
-N CNI-ADMIN
-N CNI-FORWARD
-A FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-FORWARD
-A CNI-FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-ADMIN
-A CNI-FORWARD -d 172.18.0.234/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A CNI-FORWARD -s 172.18.0.234/32 -j ACCEPT
-A CNI-FORWARD -d 172.18.0.235/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A CNI-FORWARD -s 172.18.0.235/32 -j ACCEPT
-A CNI-FORWARD -d 172.18.0.239/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A CNI-FORWARD -s 172.18.0.239/32 -j ACCEPT

@stealthybox stealthybox requested a review from twelho as a code owner September 19, 2019 17:51
@stealthybox stealthybox added the area/networking Issues related to networking label Sep 19, 2019
@chanwit chanwit requested review from chanwit and removed request for twelho September 20, 2019 06:56
@chanwit chanwit added this to the v0.6.1 milestone Sep 20, 2019
chanwit
chanwit approved these changes Sep 20, 2019
View reviewed changes
pkg/network/cni/cni.go
@@ -59,6 +59,9 @@ var igniteCNIConf = fmt.Sprintf(`{
"capabilities": {
"portMappings": true
}
},
{
"type": "firewall"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's just that simple?
I'm surprised!!

@chanwit
Copy link
Member

chanwit commented Sep 20, 2019

It worked nicely.

./ignite-test.sh 
containerd cni
INFO[0006] Created VM with ID "0c9b0fd4a2e88556" and name "421_1" 
INFO[0009] Networking is handled by "cni"               
INFO[0009] Started Firecracker VM "0c9b0fd4a2e88556" in a container with ID "ignite-0c9b0fd4a2e88
556" 
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
64 bytes from 8.8.4.4: icmp_seq=1 ttl=47 time=63.7 ms
containerd cni
INFO[0007] Created VM with ID "584d8632dac523d2" and name "421_2" 
INFO[0008] Networking is handled by "cni"               
INFO[0008] Started Firecracker VM "584d8632dac523d2" in a container with ID "ignite-584d8632dac52
3d2" 
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
64 bytes from 8.8.4.4: icmp_seq=1 ttl=47 time=102 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=47 time=96.3 ms

--- 8.8.4.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 103ms
rtt min/avg/max/mdev = 96.338/99.535/102.733/3.213 ms

and iptables

sudo iptables-save | grep CNI
:CNI-151a7366721bd2b28a9c7289 - [0:0]
:CNI-25a14a18e9a9641f37762079 - [0:0]
-A POSTROUTING -s 172.18.0.28/32 -m comment --comment "name: \"ignite-containerd-bridge\" id: \"ignite-0c9b0fd4a2e88556\"" -j CNI-25a14a18e9a9641f37762079
-A POSTROUTING -s 172.18.0.29/32 -m comment --comment "name: \"ignite-containerd-bridge\" id: \"ignite-584d8632dac523d2\"" -j CNI-151a7366721bd2b28a9c7289
-A CNI-151a7366721bd2b28a9c7289 -d 172.18.0.0/16 -m comment --comment "name: \"ignite-containerd-bridge\" id: \"ignite-584d8632dac523d2\"" -j ACCEPT
-A CNI-151a7366721bd2b28a9c7289 ! -d 224.0.0.0/4 -m comment --comment "name: \"ignite-containerd-bridge\" id: \"ignite-584d8632dac523d2\"" -j MASQUERADE
-A CNI-25a14a18e9a9641f37762079 -d 172.18.0.0/16 -m comment --comment "name: \"ignite-containerd-bridge\" id: \"ignite-0c9b0fd4a2e88556\"" -j ACCEPT
-A CNI-25a14a18e9a9641f37762079 ! -d 224.0.0.0/4 -m comment --comment "name: \"ignite-containerd-bridge\" id: \"ignite-0c9b0fd4a2e88556\"" -j MASQUERADE
:CNI-ADMIN - [0:0]
:CNI-FORWARD - [0:0]
-A FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-FORWARD
-A CNI-FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-ADMIN
-A CNI-FORWARD -d 172.18.0.28/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A CNI-FORWARD -s 172.18.0.28/32 -j ACCEPT
-A CNI-FORWARD -d 172.18.0.29/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A CNI-FORWARD -s 172.18.0.29/32 -j ACCEPT

@chanwit
Copy link
Member

chanwit commented Sep 20, 2019

Also cleanup done nicely after VMs was removed.

$ sudo bin/ignite rm -f 0c9 584
INFO[0000] Removing the container with ID "ignite-0c9b0fd4a2e88556" from the "cni" network 
INFO[0000] Removed VM with name "421_1" and ID "0c9b0fd4a2e88556" 
INFO[0000] Removing the container with ID "ignite-584d8632dac523d2" from the "cni" network 
INFO[0001] Removed VM with name "421_2" and ID "584d8632dac523d2" 
$ sudo iptables-save | grep CNI
:CNI-ADMIN - [0:0]
:CNI-FORWARD - [0:0]
-A FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-FORWARD
-A CNI-FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-ADMIN

@chanwit chanwit merged commit 7083e1c into weaveworks:master Sep 20, 2019
@chanwit chanwit mentioned this pull request Sep 20, 2019
Closed
@stealthybox stealthybox deleted the cni-firewall branch September 20, 2019 19:26
@stealthybox stealthybox mentioned this pull request Sep 20, 2019
Closed
@lukemarsden lukemarsden mentioned this pull request May 4, 2021
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
1 more reviewer

@chanwit chanwit chanwit approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
area/networking Issues related to networking
Projects
None yet
Milestone
v0.6.1
Development

Successfully merging this pull request may close these issues.

When using containerd + CNI network, the VM could not ping the Internet IPs Networking: no route to host
2 participants
@stealthybox @chanwit