BGP Control Plane + Native Routing + LB-IPAM + BPF masquarade results in unable to accept IPv6 traffic on LoadBalancer (1.14.0-rc.0) #26816
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
I tried to create a multi-stack load-balancer but it's unable to accept traffic on IPv6 despite the allocation of it's IP seems to have worked.
Cilium Version
cilium-cli: v0.12.1 compiled with go1.18.5 on linux/amd64
cilium image (default): v1.12.0
cilium image (stable): v1.13.4
cilium image (running): v1.14.0-rc.0
Kernel Version
5.19.0-1022-raspi
5.19.0-46-generic
6.1.0-10-amd64
6.1.38-060138-generic
Kubernetes Version
{
"clientVersion": {
"major": "1",
"minor": "27",
"gitVersion": "v1.27.2",
"gitCommit": "7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647",
"gitTreeState": "archive",
"buildDate": "2023-07-03T16:23:17Z",
"goVersion": "go1.20.5",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v5.0.1",
"serverVersion": {
"major": "1",
"minor": "26",
"gitVersion": "v1.26.3+k3s1",
"gitCommit": "01ea3ff27be0b04f945179171cec5a8e11a14f7b",
"gitTreeState": "clean",
"buildDate": "2023-03-27T22:23:17Z",
"goVersion": "go1.19.7",
"compiler": "gc",
"platform": "linux/amd64"
}
}Sysdump
cilium-sysdump-20230714-013030.zip
Relevant log output
# operator
level=error msg="service upsert failed" error="handleUpsertService: stripOrImportIngresses: Error while attempting to allocate IP '2001:67c:1104:fdb::'" subsys=lbipam
level=error msg="service upsert failed" error="handleUpsertService: stripOrImportIngresses: Error while attempting to allocate IP '2001:67c:1104:fdb::'" subsys=lbipam
level=error msg="service upsert failed" error="handleUpsertService: stripOrImportIngresses: Error while attempting to allocate IP '2001:67c:1104:fdb::'" subsys=lbipam
# agent
evel=debug msg="create Destination" Nlri="2001:67c:1104:fdb::4443/128" Topic=Table asn=213021 component=gobgp.BgpServerInstance subsys=bgp-control-plane
level=debug msg="Restoring service" serviceID=406 serviceIP="[2001:67c:1104:fdb::4443]:80" subsys=service
level=debug msg="Restoring service" l3n4Addr="{AddrCluster:2001:67c:1104:fdb::4443 L4Addr:{Protocol:NONE Port:80} Scope:0}" subsys=service
level=debug msg="Restoring service" serviceID=407 serviceIP="[2001:67c:1104:fdb::4443]:443" subsys=service
level=debug msg="Restoring service" l3n4Addr="{AddrCluster:2001:67c:1104:fdb::4443 L4Addr:{Protocol:NONE Port:443} Scope:0}" subsys=service
level=debug msg="Upserting service" backends="[]" l7LBFrontendPorts="[]" l7LBProxyPort=0 loadBalancerSourceRanges="[]" serviceIP="{2001:67c:1104:fdb::4443 {TCP 80} 0}" serviceName=ingress-nginx-controller serviceNamespace=networking sessionAffinity=false sessionAffinityTimeout=0 subsys=service svcExtTrafficPolicy=Cluster svcHealthCheckNodePort=0 svcIntTrafficPolicy=Cluster svcType=LoadBalancer
level=debug msg="Acquired service ID" backends="[]" l7LBFrontendPorts="[]" l7LBProxyPort=0 loadBalancerSourceRanges="[]" serviceID=406 serviceIP="{2001:67c:1104:fdb::4443 {TCP 80} 0}" serviceName=ingress-nginx-controller serviceNamespace=networking sessionAffinity=false sessionAffinityTimeout=0 subsys=service svcExtTrafficPolicy=Cluster svcHealthCheckNodePort=0 svcIntTrafficPolicy=Cluster svcType=LoadBalancer
level=debug msg="Upserted service entry" backendSlot=0 subsys=map-lb svcKey="[2001:67c:1104:fdb::4443]:20480" svcVal="0 0 (38401) [0x60 0x0]"
level=debug msg="Upserting service" backends="[]" l7LBFrontendPorts="[]" l7LBProxyPort=0 loadBalancerSourceRanges="[]" serviceIP="{2001:67c:1104:fdb::4443 {TCP 443} 0}" serviceName=ingress-nginx-controller serviceNamespace=networking sessionAffinity=false sessionAffinityTimeout=0 subsys=service svcExtTrafficPolicy=Cluster svcHealthCheckNodePort=0 svcIntTrafficPolicy=Cluster svcType=LoadBalancer
level=debug msg="Acquired service ID" backends="[]" l7LBFrontendPorts="[]" l7LBProxyPort=0 loadBalancerSourceRanges="[]" serviceID=407 serviceIP="{2001:67c:1104:fdb::4443 {TCP 443} 0}" serviceName=ingress-nginx-controller serviceNamespace=networking sessionAffinity=false sessionAffinityTimeout=0 subsys=service svcExtTrafficPolicy=Cluster svcHealthCheckNodePort=0 svcIntTrafficPolicy=Cluster svcType=LoadBalancer
level=debug msg="Upserted service entry" backendSlot=0 subsys=map-lb svcKey="[2001:67c:1104:fdb::4443]:47873" svcVal="0 0 (38657) [0x60 0x0]"
level=debug msg="Upserting service" backends="[[fddf:f7bc:9670:2::42be]:80 [fddf:f7bc:9670:3::f814]:80]" l7LBFrontendPorts="[]" l7LBProxyPort=0 loadBalancerSourceRanges="[]" serviceIP="{2001:67c:1104:fdb::4443 {TCP 80} 0}" serviceName=ingress-nginx-controller serviceNamespace=networking sessionAffinity=false sessionAffinityTimeout=0 subsys=service svcExtTrafficPolicy=Cluster svcHealthCheckNodePort=0 svcIntTrafficPolicy=Cluster svcType=LoadBalancer
level=debug msg="Acquired service ID" backends="[[fddf:f7bc:9670:2::42be]:80 [fddf:f7bc:9670:3::f814]:80]" l7LBFrontendPorts="[]" l7LBProxyPort=0 loadBalancerSourceRanges="[]" serviceID=406 serviceIP="{2001:67c:1104:fdb::4443 {TCP 80} 0}" serviceName=ingress-nginx-controller serviceNamespace=networking sessionAffinity=false sessionAffinityTimeout=0 subsys=service svcExtTrafficPolicy=Cluster svcHealthCheckNodePort=0 svcIntTrafficPolicy=Cluster svcType=LoadBalancer
level=debug msg="Upserted service entry" backendSlot=1 subsys=map-lb svcKey="[2001:67c:1104:fdb::4443]:20480" svcVal="1708 0 (38401) [0x0 0x0]"
level=debug msg="Upserted service entry" backendSlot=2 subsys=map-lb svcKey="[2001:67c:1104:fdb::4443]:20480" svcVal="1709 0 (38401) [0x0 0x0]"
level=debug msg="Upserted service entry" backendSlot=0 subsys=map-lb svcKey="[2001:67c:1104:fdb::4443]:20480" svcVal="0 2 (38401) [0x60 0x0]"
level=debug msg="Upserting service" backends="[[fddf:f7bc:9670:2::42be]:443 [fddf:f7bc:9670:3::f814]:443]" l7LBFrontendPorts="[]" l7LBProxyPort=0 loadBalancerSourceRanges="[]" serviceIP="{2001:67c:1104:fdb::4443 {TCP 443} 0}" serviceName=ingress-nginx-controller serviceNamespace=networking sessionAffinity=false sessionAffinityTimeout=0 subsys=service svcExtTrafficPolicy=Cluster svcHealthCheckNodePort=0 svcIntTrafficPolicy=Cluster svcType=LoadBalancer
level=debug msg="Acquired service ID" backends="[[fddf:f7bc:9670:2::42be]:443 [fddf:f7bc:9670:3::f814]:443]" l7LBFrontendPorts="[]" l7LBProxyPort=0 loadBalancerSourceRanges="[]" serviceID=407 serviceIP="{2001:67c:1104:fdb::4443 {TCP 443} 0}" serviceName=ingress-nginx-controller serviceNamespace=networking sessionAffinity=false sessionAffinityTimeout=0 subsys=service svcExtTrafficPolicy=Cluster svcHealthCheckNodePort=0 svcIntTrafficPolicy=Cluster svcType=LoadBalancer
level=debug msg="Upserted service entry" backendSlot=1 subsys=map-lb svcKey="[2001:67c:1104:fdb::4443]:47873" svcVal="1710 0 (38657) [0x0 0x0]"
level=debug msg="Upserted service entry" backendSlot=2 subsys=map-lb svcKey="[2001:67c:1104:fdb::4443]:47873" svcVal="1711 0 (38657) [0x0 0x0]"
level=debug msg="Upserted service entry" backendSlot=0 subsys=map-lb svcKey="[2001:67c:1104:fdb::4443]:47873" svcVal="0 2 (38657) [0x60 0x0]"
level=debug msg="sent update" Key="2001:14ba:16fd:961c::1" State=BGP_FSM_ESTABLISHED Topic=Peer asn=213021 attributes="[{Origin: i} {LocalPref: 100} {MpReach(ipv6-unicast): {Nexthop: 2001:14ba:16fd:961c:ee8e:b5ff:fe7b:efaa, NLRIs: [2001:67c:1104:fdb::4443/128]}}]" component=gobgp.BgpServerInstance nlri="[]" subsys=bgp-control-plane withdrawals="[]"Anything else?
Current Helm values: https://github.com/samip5/k8s-cluster/blob/22b9c7c6448ad2c6164b9891a73c82320636bc9a/cluster/apps/cilium-system/cilium/app/helm-release.yaml
Output of cilium-health status is available at: https://bpa.st/6VEP2 (Seems to be that endpoint connectivity via IPv6 is not working, but is that the issue?)
I can see the packet arriving at the node:
02:03:35.412490 IP6 (flowlabel 0xde09b, hlim 63, next-header unknown (60) payload length: 68) 2001:14ba:16fd:9610:2468:bebf:3024:a6d0 > fddf:f7bc:9670:2::42be: DSTOPT (opt_type 0x1b: len=20) 51275 > 443: Flags [SEW], cksum 0x1072 (correct), seq 558173269, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 2895910638 ecr 0,sackOK,eol], length 0
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
Impacts bpf/ or low-level forwarding details, including map management and monitor messages.Relates to IPv6 protocol supportRelates to SNAT or Masquerading of trafficThis is a bug in the Cilium logic.This was reported by a user in the Cilium community, eg via Slack.This issue requires triaging to establish severity and next steps.