cilium-ingress of service type NodePort doesn't work for external wireguard interface #26584
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
I have a single node k3s cluster with external/self-managed wireguard interface.
KubeProxyReplacement: True [eth0 192.168.1.110 fd7a:115c:a1e0:ab12:4843:cd96:6279:2c75, tailscale0 100.121.44.117 fd7a:115c:a1e0:ab12:4843:cd96:6279:2c75, wg0 172.20.24.10 fd00:ebee:172:2024::10 (Direct Routing), wlan0 192.168.1.109 fd7a:115c:a1e0:ab12:4843:cd96:6279:2c75]
Yesterday I tried to use the cilium ingress controller feature. I created a shared cilium-ingress of service type NodePort:
ingressController:
enabled: true
loadbalancerMode: shared
service:
insecureNodePort: 30080
secureNodePort: 30443
type: NodePortkube-system cilium-ingress NodePort 10.43.185.167 <none> 80:30080/TCP,443:30443/TCP 21h
I can access the above nodeport using eth0/wlan0 or wg0 ip address on the same node.
$ curl 172.20.24.10:30443 -i
HTTP/1.1 404 Not Found
date: Sat, 01 Jul 2023 07:19:35 GMT
server: envoy
content-length: 0
But on a different host, I can't access the nodeport using the wg0 ip address, the eth0/wlan0 works as expected.
$ curl 172.20.24.10:30443 -i
curl: (28) Failed to connect to 172.20.24.10 port 30443 after 75007 ms: Couldn't connect to server
I haven't noticed any packet drop in hubble-ui and pwru. And the normal NodePort service works with wg0 on a different node:
$ curl 172.20.24.10:30761 -I
HTTP/1.1 200 OK
Server: nginx/1.22.0
Date: Sat, 01 Jul 2023 07:28:04 GMT
Content-Type: text/html
Content-Length: 785
Last-Modified: Fri, 02 Jun 2023 11:36:40 GMT
Connection: keep-alive
ETag: "6479d448-311"
Cache-Control: no-store, no-cache, must-revalidate
Accept-Ranges: bytes
0xffff9a0802b34100 2 [kworker/2:3-wg-crypt-wg0] skb_release_head_state netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=64 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0802b34100 2 [kworker/2:3-wg-crypt-wg0] skb_release_data netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=64 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0802b34100 2 [kworker/2:3-wg-crypt-wg0] skb_free_head netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=64 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0802b34100 2 [kworker/2:3-wg-crypt-wg0] kfree_skbmem netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=64 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] tcp4_gro_receive netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] tcp_gro_receive netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] skb_defer_rx_timestamp netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] tpacket_rcv netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] tpacket_get_timestamp netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] consume_skb netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] tcf_classify netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] skb_ensure_writable netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
0xffff9a0938238600 0 [kworker/0:7-wg-crypt-wg0] kfree_skb_reason(SKB_DROP_REASON_TC_INGRESS) netns=4026531840 mark=0x0 ifindex=5 proto=8 mtu=1420 len=48 172.20.27.10:56780->172.20.24.10:30080(tcp)
Cilium Version
cilium-cli: 0.14.3 compiled with go1.20.5 on linux/amd64
cilium image (default): v1.13.2
cilium image (stable): unknown
cilium image (running): v1.14.0-rc.0
Kernel Version
6.1.0-kali9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1kali1 (2023-05-12) x86_64 GNU/Linux
Kubernetes Version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"archive", BuildDate:"1980-01-01T00:00:00Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2+k3s1", GitCommit:"213d7ad499e166290872f51c63d8eaa2f1fe78b3", GitTreeState:"clean", BuildDate:"2023-05-26T22:14:11Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
Sysdump
cilium-sysdump-20230701-153111.zip
Relevant log output
No response
Anything else?
No response
Code of Conduct
- I agree to follow this project's Code of Conduct