Skip to content

IPv6 not working between nodes because of extra route #28327

@CallMeFoxie

Description

@CallMeFoxie

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

With cilium 1.14 we are having troubles with IPv6 routing between nodes. Cilium adds this to the interface:

7: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9a:aa:d1:1e:5c:dc brd ff:ff:ff:ff:ff:ff
    inet 10.11.22.33/32 scope global cilium_host
       valid_lft forever preferred_lft forever
    inet6 2a02:aa:bb:8a00::2080:2bd/64 scope global 
       valid_lft forever preferred_lft forever

which also - by default - adds a route

2a02:aa:bb:8a00::2080:2b8 dev lxc_health proto kernel metric 1024 pref medium
2a02:aa:bb:8a00::2080:200/120 dev cilium_host proto kernel src 2a02:aa:bb:8a00::2080:2bd metric 1024 pref medium
2a02:aa:bb:8a00::/64 dev cilium_host proto kernel metric 256 pref medium

which crosses with our ip pool:

2a02:aa:bb:8a00::2080:0/105

meaning all the pod-to-pod traffic gets improperly routed.

It seems to work fine with Cilium 1.13. I do not have any kubernetes cluster by hand to check, but our openstack cilium 1.13 cluster has assigned IPv6 with /128 mask but also --ipv6-node flag set.

There's noprefixroute flag in netlink which seems to add the IP but not add the route on that interface, but I am struggling to build cilium outside of CI and it is Friday ( 🙃 ) so cannot test it until Monday. My idea to try out is this patch:

diff --git a/pkg/datapath/loader/netlink.go b/pkg/datapath/loader/netlink.go
index bfecce37f0..9b1ce01ced 100644
--- a/pkg/datapath/loader/netlink.go
+++ b/pkg/datapath/loader/netlink.go
@@ -435,6 +435,7 @@ func addHostDeviceAddr(hostDev netlink.Link, ipv4, ipv6 net.IP) error {
                                IP:   ipv6,
                                Mask: net.CIDRMask(64, 128), // corresponds to /64
                        },
+                       Flags: unix.IFA_F_NOPREFIXROUTE,
                }
 
                if err := netlink.AddrReplace(hostDev, &addr); err != nil {

if this works I will be happy to sent a PR

Cilium Version

1.14.1

Kernel Version

5.15.0-69-generic

Kubernetes Version

1.27.1

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.feature/ipv6Relates to IPv6 protocol supportkind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions