inconsistent behavior with forceDeviceDetection #37425
Description
Is there an existing issue for this?
- I have searched the existing issues
Version
equal or higher than v1.16.6 and lower than v1.17.0
What happened?
I am observing an inconsistency when enabling the Helm flag forceDeviceDetection when used with the devices flag:
-
It appears that in the default configuration, when not setting either flag, Cilium's device auto-detection feature is looking for a configured IPv4 address on an interface.
-
When
devicesandforceDeviceDetectionare set, it appears that the presence of an IPv6 address of any kind is the only evaluation criterion. This is causing more interfaces to be picked up than when not set.
In our testing, the default configuration on one node selected:
KubeProxyReplacement: True [enp193s0np0 10.176.207.201 fe80::a288:c2ff:fe36:2742 (Direct Routing)]
Masquerading: BPF [enp193s0np0] 10.96.0.0/13 [IPv4: Enabled, IPv6: Disabled]
However, when configuring:
devices=e+
forceDeviceDetection=true
on the same node, it selected:
KubeProxyReplacement: True [enp193s0np0 10.176.207.67 fe80::a288:c2ff:fe46:9eea (Direct Routing), enp193s0v0 fe80::f055:a1ff:febe:7428, enp193s0v1 fe80::104b:99ff:fe26:cd3d, enp193s0v2 fe80::bc11:5fff:fe55:9bcf, enp193s0v3 fe80::90a4:a9ff:fecc:c770, enp193s0v4 fe80::33:7aff:fef5:3b50, enp193s0v5 fe80::bd:c5ff:fe0c:1eee, enp193s0v6 fe80::a8dd:f7ff:feb0:5f8c, enp193s0v7 fe80::14d7:f0ff:fe34:a55f]
Masquerading: BPF [enp193s0np0, enp193s0v0, enp193s0v1, enp193s0v2, enp193s0v3, enp193s0v4, enp193s0v5, enp193s0v6, enp193s0v7] 10.96.0.0/13 [IPv4: Enabled, IPv6: Disabled]
Context
I am trying to configure these options because, with the default configuration, Cilium's autodetection feature is selecting Infiniband interfaces. When an IB interface is selected, it results in errors in Cilium's logs and a false positive to the Prometheus metric controllers_failing. For the IB interfaces that were selected, all had an IP address configured. Though not as common, there is a valid use case in IB with an IP address on the interface.
This is taking us down the path of specifying e+ as a match for devices, to avoid caring about specific interface names. Our Infiband interface names begin with ib.
How can we reproduce the issue?
The interfaces and their IP addresses on the node where we were testing forceDeviceDection were:
Notes:
enp193s0np0is the primary interface and the only one which we intended for Cilium to use.- all others ending in
v#appear to be unused virtual interfaces.
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp193s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether a0:88:c2:46:9e:ea brd ff:ff:ff:ff:ff:ff
inet 10.176.207.67/31 metric 1024 scope global dynamic enp193s0np0
valid_lft 540893sec preferred_lft 540893sec
inet6 fe80::a288:c2ff:fe46:9eea/64 scope link
valid_lft forever preferred_lft forever
3: enp193s0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5173 qdisc mq state UP group default qlen 1000
link/ether 0a:b0:cf:43:c2:e8 brd ff:ff:ff:ff:ff:ff permaddr 66:25:22:ec:55:bd
inet6 fe80::f055:a1ff:febe:7428/64 scope link
valid_lft forever preferred_lft forever
4: enp193s0v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5177 qdisc mq state UP group default qlen 1000
link/ether 0a:b0:cf:43:82:0f brd ff:ff:ff:ff:ff:ff permaddr 3e:7c:c8:13:73:2f
inet6 fe80::104b:99ff:fe26:cd3d/64 scope link
valid_lft forever preferred_lft forever
5: enp193s0v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5185 qdisc mq state UP group default qlen 1000
link/ether 0a:b0:cf:43:95:fa brd ff:ff:ff:ff:ff:ff permaddr ae:f1:6c:67:31:54
inet6 fe80::bc11:5fff:fe55:9bcf/64 scope link
valid_lft forever preferred_lft forever
6: enp193s0v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5191 qdisc mq state UP group default qlen 1000
link/ether 0a:b0:cf:43:18:27 brd ff:ff:ff:ff:ff:ff permaddr aa:50:7b:7a:98:d2
inet6 fe80::90a4:a9ff:fecc:c770/64 scope link
valid_lft forever preferred_lft forever
7: enp193s0v4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5195 qdisc mq state UP group default qlen 1000
link/ether 0a:b0:cf:43:3c:40 brd ff:ff:ff:ff:ff:ff permaddr de:10:1f:7a:25:9a
inet6 fe80::33:7aff:fef5:3b50/64 scope link
valid_lft forever preferred_lft forever
8: enp193s0v5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5200 qdisc mq state UP group default qlen 1000
link/ether 0a:b0:cf:43:06:4d brd ff:ff:ff:ff:ff:ff permaddr 6e:82:bf:00:3c:19
inet6 fe80::bd:c5ff:fe0c:1eee/64 scope link
valid_lft forever preferred_lft forever
9: enp193s0v6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5206 qdisc mq state UP group default qlen 1000
link/ether 0a:b0:cf:43:fc:84 brd ff:ff:ff:ff:ff:ff permaddr b6:0f:2b:38:f7:9d
inet6 fe80::a8dd:f7ff:feb0:5f8c/64 scope link
valid_lft forever preferred_lft forever
10: enp193s0v7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5214 qdisc mq state UP group default qlen 1000
link/ether 0a:b0:cf:43:f1:3d brd ff:ff:ff:ff:ff:ff permaddr ca:f1:e7:cb:b7:b6
inet6 fe80::14d7:f0ff:fe34:a55f/64 scope link
valid_lft forever preferred_lft forever
11: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether be:cd:b5:12:62:d3 brd ff:ff:ff:ff:ff:ff
inet6 fe80::bccd:b5ff:fe12:62d3/64 scope link
valid_lft forever preferred_lft forever
12: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 2e:f3:6a:92:f9:7e brd ff:ff:ff:ff:ff:ff
inet 10.96.0.228/32 scope global cilium_host
valid_lft forever preferred_lft forever
inet6 fe80::2cf3:6aff:fe92:f97e/64 scope link
valid_lft forever preferred_lft forever
16: lxccc2997b127a3@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether ee:33:b2:c4:b1:ca brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::ec33:b2ff:fec4:b1ca/64 scope link
valid_lft forever preferred_lft forever
18: lxcb8178515c502@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 6e:36:61:85:34:47 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::6c36:61ff:fe85:3447/64 scope link
valid_lft forever preferred_lft forever
20: lxc0642b1cd6cd0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 52:29:a0:78:e7:94 brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::5029:a0ff:fe78:e794/64 scope link
valid_lft forever preferred_lft forever
22: lxc5f8057e2d119@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 02:90:77:73:4e:b6 brd ff:ff:ff:ff:ff:ff link-netnsid 4
inet6 fe80::90:77ff:fe73:4eb6/64 scope link
valid_lft forever preferred_lft forever
24: lxc22cc0f11476e@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 5a:2e:01:0b:c9:4d brd ff:ff:ff:ff:ff:ff link-netnsid 5
inet6 fe80::582e:1ff:fe0b:c94d/64 scope link
valid_lft forever preferred_lft forever
26: lxc1221c4ff0dea@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 0e:f1:77:04:10:07 brd ff:ff:ff:ff:ff:ff link-netnsid 6
inet6 fe80::cf1:77ff:fe04:1007/64 scope link
valid_lft forever preferred_lft forever
30: lxc7e19f740a5e0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 0e:fa:14:ed:a4:c1 brd ff:ff:ff:ff:ff:ff link-netnsid 8
inet6 fe80::cfa:14ff:feed:a4c1/64 scope link
valid_lft forever preferred_lft forever
32: lxc723c99e81f61@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 56:8b:3c:5f:a4:05 brd ff:ff:ff:ff:ff:ff link-netnsid 9
inet6 fe80::548b:3cff:fe5f:a405/64 scope link
valid_lft forever preferred_lft forever
34: lxc71dcb55a0208@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 6e:57:a8:8b:ca:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 11
inet6 fe80::6c57:a8ff:fe8b:cac6/64 scope link
valid_lft forever preferred_lft forever
36: lxce30b52889049@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether c6:a8:dd:ee:42:de brd ff:ff:ff:ff:ff:ff link-netnsid 10
inet6 fe80::c4a8:ddff:feee:42de/64 scope link
valid_lft forever preferred_lft forever
38: lxcd70888cded01@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether f6:8a:00:f2:d3:a7 brd ff:ff:ff:ff:ff:ff link-netnsid 12
inet6 fe80::f48a:ff:fef2:d3a7/64 scope link
valid_lft forever preferred_lft forever
40: lxc119762dab1e2@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 7e:f7:af:14:de:b1 brd ff:ff:ff:ff:ff:ff link-netnsid 13
inet6 fe80::7cf7:afff:fe14:deb1/64 scope link
valid_lft forever preferred_lft forever
42: lxcc748237fa556@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 76:11:66:b1:3e:3e brd ff:ff:ff:ff:ff:ff link-netnsid 14
inet6 fe80::7411:66ff:feb1:3e3e/64 scope link
valid_lft forever preferred_lft forever
44: lxc2a231692aa43@if43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 36:bb:a7:85:96:30 brd ff:ff:ff:ff:ff:ff link-netnsid 15
inet6 fe80::34bb:a7ff:fe85:9630/64 scope link
valid_lft forever preferred_lft forever
46: lxc9c7765dc9fe6@if45: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 2e:0f:8e:f5:7b:f7 brd ff:ff:ff:ff:ff:ff link-netnsid 16
inet6 fe80::2c0f:8eff:fef5:7bf7/64 scope link
valid_lft forever preferred_lft forever
48: lxc470aeef967ad@if47: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 5a:f0:dd:9d:bd:ab brd ff:ff:ff:ff:ff:ff link-netnsid 17
inet6 fe80::58f0:ddff:fe9d:bdab/64 scope link
valid_lft forever preferred_lft forever
50: lxc218f55260401@if49: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 7a:cd:47:28:9d:10 brd ff:ff:ff:ff:ff:ff link-netnsid 18
inet6 fe80::78cd:47ff:fe28:9d10/64 scope link
valid_lft forever preferred_lft forever
52: lxc43db3386ca8c@if51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 9e:4f:cc:d3:89:7d brd ff:ff:ff:ff:ff:ff link-netnsid 19
inet6 fe80::9c4f:ccff:fed3:897d/64 scope link
valid_lft forever preferred_lft forever
54: lxccf468c59638b@if53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 9e:7f:1c:45:32:0f brd ff:ff:ff:ff:ff:ff link-netnsid 21
inet6 fe80::9c7f:1cff:fe45:320f/64 scope link
valid_lft forever preferred_lft forever
56: lxc927aef141af8@if55: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether c6:e7:2a:6b:e9:ac brd ff:ff:ff:ff:ff:ff link-netnsid 20
inet6 fe80::c4e7:2aff:fe6b:e9ac/64 scope link
valid_lft forever preferred_lft forever
58: lxc265bc0de0414@if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether ca:bf:46:db:17:30 brd ff:ff:ff:ff:ff:ff link-netnsid 22
inet6 fe80::c8bf:46ff:fedb:1730/64 scope link
valid_lft forever preferred_lft forever
60: lxc63cd7dc8d655@if59: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether ca:e1:95:c3:3b:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 7
inet6 fe80::c8e1:95ff:fec3:3be8/64 scope link
valid_lft forever preferred_lft forever
66: lxc_health@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 02:15:d4:ad:65:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::15:d4ff:fead:65e7/64 scope link
valid_lft forever preferred_lft forever
To reproduce this issue, start with a single network interface configured in a valid configuration with Cilium installed.
Then, add a virtual network interface with a name that started with e, and enable IPv6 on it but configure no addresses. This should result in an IPv6 Link Local address being configured.
Before changing Cilium's configuration, take note of the interfaces selected. It is expected that the virtual address added just before not be selected. This is because it lacks a routable IPv4 address.
Next, modify Cilium helm chart and/or cilium-agent configuration to set:
devices=e+
forceDeviceDetection=true
Observe the difference. Given this report, it is expected that the virtual device lacking a routable IP be selected even though it is unsusable.
When fixed, the expected behavior is that the virtual device lacking a routable IP continue to NOT be selected.
Cilium Version
Client: 1.16.1 6857905 2024-08-13T13:29:59+00:00 go version go1.22.5 linux/amd64
Daemon: 1.16.1 6857905 2024-08-13T13:29:59+00:00 go version go1.22.5 linux/amd64
Kernel Version
Will update
Kubernetes Version
Will update
Regression
No response
Sysdump
No response
Relevant log output
Anything else?
No response
Cilium Users Document
- Are you a user of Cilium? Please add yourself to the Users doc
Code of Conduct
- I agree to follow this project's Code of Conduct