Cilium non-deterministically classifies CIDR policy matches for range with node IPs #16308
Description
Bug report
General Information
- Cilium version: cilium 1.9.4
- Kernel version: Linux ip-10-29-109-25 5.4.0-1045-aws Adding labels to ui #47-Ubuntu SMP Tue Apr 13 07:02:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- Orchestration system version in use: kubernetes v1.20.6
How to reproduce the issue
Run node-local-dns --setupiptables=false and kubelet --cluster-dns=169.254.20.10
Add a CNP that looks like this:
spec:
egress:
- toCIDR:
- 169.254.20.10/32
toPorts:
- ports:
- port: "53"
protocol: ANY
endpointSelector: {}
On first cilium run, cilium will deny traffic to node-local-dns. If the pod is deleted, the second cilium run will accept traffic.
First cilium run
cilium ip list:
169.254.20.10/32 host
cilium monitor -t policy-verdict:
Policy verdict log: flow 0xaaa04ad local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:53076 -> 169.254.20.10:53 udp
Policy verdict log: flow 0xaaa04ad local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:53076 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x0 local EP ID 481, remote ID 17956, proto 6, ingress, action deny, match none, 100.71.139.54:45062 -> 100.71.181.177:6943 tcp SYN
Policy verdict log: flow 0xaaa04ad local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:53076 -> 169.254.20.10:53 udp
Policy verdict log: flow 0xaaa04ad local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:53076 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x897b78c4 local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:56843 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x897b78c4 local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:56843 -> 169.254.20.10:53 udp
Second cilium run
169.254.20.10/32 cidr:169.254.0.0/16
cidr:169.254.0.0/19
cidr:169.254.20.10/31
cidr:168.0.0.0/5
cidr:168.0.0.0/7
cidr:169.254.0.0/17
cidr:169.254.20.0/23
reserved:world
cidr:169.254.20.0/22
cidr:169.254.20.10/32
cidr:169.254.20.0/25
cidr:169.254.20.0/27
cidr:169.0.0.0/8
cidr:169.128.0.0/9
cidr:169.254.0.0/15
cidr:169.224.0.0/11
cidr:168.0.0.0/6
cidr:169.254.20.0/26
cidr:160.0.0.0/3
cidr:169.254.16.0/20
cidr:0.0.0.0/0
cidr:169.254.16.0/21
cidr:128.0.0.0/2
cidr:169.248.0.0/13
cidr:169.192.0.0/10
cidr:169.254.20.8/30
cidr:169.254.0.0/18
cidr:160.0.0.0/4
cidr:169.254.20.0/28
cidr:169.254.20.8/29
cidr:169.240.0.0/12
cidr:169.252.0.0/14
cidr:169.254.20.0/24
cidr:128.0.0.0/1
cilium monitor -t policy-verdict:
Policy verdict log: flow 0x5f42acab local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:38685 -> 169.254.20.10:53 udp
Policy verdict log: flow 0xbe339248 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:54033 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x3a516849 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:40686 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x656356b6 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:41152 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x978580ec local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:53109 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x45c1e6c1 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:50593 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x9cd850d5 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:37507 -> 169.254.20.10:53 udp
Slack thread https://cilium.slack.com/archives/C1MATJ5U5/p1621963201289500
Metadata
Metadata
Assignees
Labels
Impacts bpf/ or low-level forwarding details, including map management and monitor messages.This is a bug in the Cilium logic.This was reported by a user in the Cilium community, eg via Slack.This issue requires triaging to establish severity and next steps.These issues are not marked stale by our issue bot.Impacts whether traffic is allowed or denied based on user-defined policies.