Two VMs on 2 separate hosts, how to make networking work with Ignite and WeaveNet? #628
Description
Hello weaveworks team,
I am trying to spin up 2 independant k8s clusters, each having one respective worker node using ignite. I have 2 machines for this test:
Host-1 (IP: 192.168.68.130)
|_ K8S Master 1
|_ K8S Master 2
Host-2 (IP: 192.168.68.134)
|_ K8S Worker 1 (attache to master-1)
|_ K8S Worker 2 (attache to master-2)
Therefore, I want to set up a WeaveNet CNI network on both hosts using the instructions provided from your documentation:
# On Host-1 & Host-2
docker run -d \
--privileged \
--net host \
--pid host \
--restart always \
-e HOSTNAME="$(hostname)" \
-e KUBE_PEERS="192.168.68.130 192.168.68.134" \
-v /var/lib/weave:/weavedb \
-v /opt:/host/opt \
-v /home:/host/home \
-v /etc:/host/etc \
-v /var/lib/dbus:/host/var/lib/dbus \
-v /lib/modules:/lib/modules \
-v /run/xtables.lock:/run/xtables.lock \
--entrypoint /home/weave/launch.sh \
weaveworks/weave-kube:2.5.2I removed the file rm -rf /etc/cni/net.d/10-ignite.conflist, then started my Master-1 on Host-1, and Worker-1 on Host-2 using the following command:
# On Host-1
sudo ignite \
--network-plugin cni \
run weaveworks/ignite-kubeadm:latest \
--cpus 2 \
--memory 2GB \
--ssh \
--name master-1and
# On Host-2
sudo ignite \
--network-plugin cni \
run weaveworks/ignite-kubeadm:latest \
--cpus 2 \
--memory 2GB \
--ssh \
--name worker-1PS: I am not even attempting to start the K8S cluster here, I first want to get the networking part right on one of the two clusters.
Then I ssh into each VM using sudo ignite ssh master-1 and sudo ignite ssh worker-1 on each respective host.
I was expecting on Host-1, VM Master-1 to have a network interface called eth0 with an IP address that is part of a subnet shared with the VM on Host-2, VM Worker-2.
When I did a sudo ignite ps on both hosts, I got the following results:
VM ID IMAGE KERNEL SIZE CPUS MEMORY CREATED STATUS IPS PORTS NAME
7962498dc0401f33 weaveworks/ignite-kubeadm:latest weaveworks/ignite-kernel:4.19.125 4.0 GB 2 2.0 GB 22m ago Up 22m 10.32.0.2 master-1
and
VM ID IMAGE KERNEL SIZE CPUS MEMORY CREATED STATUS IPS PORTS NAME
a9f37cb3378c6c00 weaveworks/ignite-kubeadm:latest weaveworks/ignite-kernel:4.19.125 4.0 GB 2 2.0 GB 3m ago Up 3m 10.40.0.1 worker-1
When I ssh into do VM Master-1 on Host-1 and do a ifconfig, I get the following:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.32.0.2 netmask 255.240.0.0 broadcast 10.47.255.255
inet6 fe80::4cdc:6fff:feb8:90a6 prefixlen 64 scopeid 0x20<link>
ether 4e:dc:6f:b8:90:a6 txqueuelen 1000 (Ethernet)
RX packets 323 bytes 28590 (28.5 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 755 bytes 64154 (64.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I have a hard time doing ssh into VM Worker-1 on Host-2, it times out the first time I try, and succedes at the second attempt. Once that ssh session started, I can not do anything, the cursor is frozen, and reacts only after 30 seconds or so. I did manage to get the output of ifconfig:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.40.0.1 netmask 255.240.0.0 broadcast 10.47.255.255
inet6 fe80::2409:8aff:fef9:500d prefixlen 64 scopeid 0x20<link>
ether 26:09:8a:f9:50:0d txqueuelen 1000 (Ethernet)
RX packets 1290 bytes 69496 (69.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1126 bytes 59796 (59.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
When I try to ping Master-1 from Worker-1, I get a Host Unreachable.
My guess is that I am doing something wrong here, but what is it?
Could someone with more experience with WeaveNet and Ignite point me in the right direction please, that would be fantastic!
Thanks