Gateway set incorrectly in VMs after 256 VMs created #664
Description
So I have a machine where I've been using ignite intensively, and it's interesting that the old IPs allocated to the VMs never seem to be cleaned up:
That's not really the issue, though, although it's contributed to it. Rather, outbound network connectivity has stopped working for new VMs. It seems to have coincided with the IPs being allocated rolling over from 10.61.0.* to 10.61.1.*. Now the default gateway is being set to 10.61.1.1, whereas on a different machine where the IPs haven't rolled up to the next /24 yet, 10.61.0.1 is correctly set as the gateway. 10.61.0.1 is pingable from the VM where 10.61.1.1 is set as the gateway, but of course outbound network connectivity is broken there.
Working machine:
root@5765398384ca573b:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.61.0.1 0.0.0.0 UG 0 0 0 eth0
10.61.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
root@5765398384ca573b:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 72:9c:43:94:5c:98 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 8e:cb:88:ab:6f:b9 brd ff:ff:ff:ff:ff:ff
inet 10.61.0.70/16 brd 10.61.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::8ccb:88ff:feab:6fb9/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:6a:74:54:29 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:6aff:fe74:5429/64 scope link
valid_lft forever preferred_lft forever
Broken machine:
root@9497d3769c4b7fd8:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.61.1.1 0.0.0.0 UG 0 0 0 eth0
10.61.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
root@9497d3769c4b7fd8:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether b6:bf:f0:8b:15:61 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 46:72:d2:1e:70:97 brd ff:ff:ff:ff:ff:ff
inet 10.61.1.9/16 brd 10.61.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::4472:d2ff:fe1e:7097/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:ca:5c:f0:d0 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
CNI config is identical on both machines:
$ cat /etc/cni/net.d/10-ignite.conflist
{
"cniVersion": "0.4.0",
"name": "ignite-cni-bridge",
"plugins": [
{
"type": "bridge",
"bridge": "ignite0",
"isGateway": true,
"isDefaultGateway": true,
"promiscMode": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.61.0.0/16"
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
},
{
"type": "firewall"
}
]
}
Something, somewhere must be seeing the VM IP set to 10.61.1.9 and assuming that the gateway is 10.61.1.1 when it's actually 10.61.0.1. It's almost like something is seeing a /16 net range and presuming it's a /24.
I've dug around the CNI networking code and can't see what's going on though, so, I raised this issue in the hope that someone smarter than me like @luxas can figure it out :-)