This repository was archived by the owner on Dec 7, 2023. It is now read-only.
This repository was archived by the owner on Dec 7, 2023. It is now read-only.
Multi host ignite VMs networking based on WeaveNet not working as expected (reopening issue #628) #642
Description
Hello WeaveWorks team,
I am reopening issue #628, the applied fix did make things slightly better but it did not fix the underlying issue.
Two Hosts, on both I run the WeaveWorks CNI docker image as described in issue #628.
I then installed ignite on both hosts and started a Ignite VM on each host with the flag --network-plugin cni respectively. On each host, the output of ifconfig is (I filtered out the other network interfaces that are not relevant to this issue):
ignite0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet 10.61.0.1 netmask 255.255.0.0 broadcast 10.61.255.255
inet6 fe80::dca5:f0ff:fed9:7481 prefixlen 64 scopeid 0x20<link>
ether de:a5:f0:d9:74:81 txqueuelen 1000 (Ethernet)
RX packets 189 bytes 17839 (17.8 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 201 bytes 20068 (20.0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethd0e92add: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::14e9:28ff:fed4:5cda prefixlen 64 scopeid 0x20<link>
ether 16:e9:28:d4:5c:da txqueuelen 0 (Ethernet)
RX packets 189 bytes 20485 (20.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 230 bytes 23476 (23.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethwe-bridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1376
inet6 fe80::ac38:4ff:feac:c4f3 prefixlen 64 scopeid 0x20<link>
ether ae:38:04:ac:c4:f3 txqueuelen 0 (Ethernet)
RX packets 196 bytes 22470 (22.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 92 bytes 10557 (10.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethwe-datapath: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1376
inet6 fe80::f8ae:8eff:feef:6077 prefixlen 64 scopeid 0x20<link>
ether fa:ae:8e:ef:60:77 txqueuelen 0 (Ethernet)
RX packets 92 bytes 10557 (10.5 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 196 bytes 22470 (22.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vxlan-6784: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 65535
inet6 fe80::f067:50ff:fe4f:45f8 prefixlen 64 scopeid 0x20<link>
ether f2:67:50:4f:45:f8 txqueuelen 1000 (Ethernet)
RX packets 240 bytes 163152 (163.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 174 bytes 155592 (155.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
weave: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1376
inet 10.32.0.1 netmask 255.240.0.0 broadcast 10.47.255.255
inet6 fe80::7cb0:f6ff:fe9e:1b0e prefixlen 64 scopeid 0x20<link>
ether 7e:b0:f6:9e:1b:0e txqueuelen 1000 (Ethernet)
RX packets 195 bytes 19650 (19.6 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 59 bytes 6808 (6.8 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
and
ignite0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet 10.61.0.1 netmask 255.255.0.0 broadcast 10.61.255.255
inet6 fe80::d8aa:29ff:fe1c:2e35 prefixlen 64 scopeid 0x20<link>
ether da:aa:29:1c:2e:35 txqueuelen 1000 (Ethernet)
RX packets 294 bytes 28890 (28.8 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 300 bytes 33022 (33.0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 552 bytes 48676 (48.6 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 552 bytes 48676 (48.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth5cf312db: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::39:f3ff:fe5e:2075 prefixlen 64 scopeid 0x20<link>
ether 02:39:f3:5e:20:75 txqueuelen 0 (Ethernet)
RX packets 294 bytes 33006 (33.0 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 330 bytes 36565 (36.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethwe-bridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1376
inet6 fe80::d47a:82ff:fe1e:5807 prefixlen 64 scopeid 0x20<link>
ether d6:7a:82:1e:58:07 txqueuelen 0 (Ethernet)
RX packets 149 bytes 15746 (15.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 111 bytes 12216 (12.2 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethwe-datapath: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1376
inet6 fe80::88f:cbff:fe49:a42b prefixlen 64 scopeid 0x20<link>
ether 0a:8f:cb:49:a4:2b txqueuelen 0 (Ethernet)
RX packets 111 bytes 12216 (12.2 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 149 bytes 15746 (15.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vxlan-6784: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 65535
inet6 fe80::a41b:37ff:fe63:e69e prefixlen 64 scopeid 0x20<link>
ether a6:1b:37:63:e6:9e txqueuelen 1000 (Ethernet)
RX packets 304 bytes 317620 (317.6 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 371 bytes 325090 (325.0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
weave: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1376
inet 10.40.0.0 netmask 255.240.0.0 broadcast 10.47.255.255
inet6 fe80::d0e2:beff:fe0c:6c35 prefixlen 64 scopeid 0x20<link>
ether d2:e2:be:0c:6c:35 txqueuelen 1000 (Ethernet)
RX packets 148 bytes 13584 (13.5 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 77 bytes 8348 (8.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I can ping 10.40.0.0 from 10.32.0.1 and vise versa, this tells me that WeaveWorks CNI is working as expected.
But when I SSH into each VM, they both have the same output for ifconfig:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.61.0.2 netmask 255.255.0.0 broadcast 10.61.255.255
inet6 fe80::3804:88ff:fec8:a6ef prefixlen 64 scopeid 0x20<link>
ether 3a:04:88:c8:a6:ef txqueuelen 1000 (Ethernet)
RX packets 303 bytes 34991 (34.9 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 215 bytes 25732 (25.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
As you can see, they both have the same IP on the WeaveWorks CNI network (obviously not what was expected), rather than each having a dedicated IP. There is no more hanging or delay, so that issue is gone now, but I still cant do multi host Ignite VM networking using WeaveNet CNI.
I also removed the file /etc/cni/net.d/10-ignite.conflist before running the ignite run command. The content of the newly created /etc/cni/net.d/10-ignite.conflist file is:
{
"cniVersion": "0.4.0",
"name": "ignite-cni-bridge",
"plugins": [
{
"type": "bridge",
"bridge": "ignite0",
"isGateway": true,
"isDefaultGateway": true,
"promiscMode": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.61.0.0/16"
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
},
{
"type": "firewall"
}
]
}Or the documentation is lacking some instructions here, or there is a bug somewhere. Could someone please have a look and help me solve this issue please?
Thanks