RKE2 with cilium and project network isolation breaks ingress controller to pod communication #34275
Description
Rancher Server Setup
- Rancher version: v2.6.0-rc3
- Installation option (Docker install/Helm Chart): Helm
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): v1.21.3+k3s1
- Proxy/Cert Details: no proxy, Let's Encrypt certificate
Information about the Cluster
- Kubernetes version: RKE2: v1.21.3+rke2r2
- Cluster Type (Local/Downstream): downstream
- If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): tested with digital ocean infrastructure provider
Describe the bug
When you create a RKE2 cluster with cilium as a CNI and activate project network isolation. The communication between the ingress controller and the Pods does not work anymore, when the ingress controller and workload pod are not on the same node.
To Reproduce
- Create a multi node RKE2 custom cluster
- Choose Cilium CNI
- Activate project network isolation
- Create a workload (e.g.) an nginx and an ingress
- Try to reach the ingress, make sure that you connect to an ingress controller Pod that is not on the same node as the nginx pod.
Result
The nginx ingress controller will get a gateway timeout when communication with the pod:
curl 35.158.100.209 -H 'Host: nginx.example'
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
Expected Result
The communication works out of the box.
Additional context
Similar issue with calico as a CNI: #33979
For cilium the host network -> pod traffic needs to be whitelisted differently as well. Upstream issue for this: cilium/cilium#10405.
A working CiliumNetworkPolicy for this:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: hn-nodes
namespace: default
spec:
endpointSelector: {}
ingress:
- fromEntities:
- remote-node