Incompatibility between Istio ambient mode readiness probes and Cilium eBPF Routing #36022
Description
Is there an existing issue for this?
- I have searched the existing issues
Version
equal or higher than v1.16.3 and lower than v1.17.0
What happened?
The bug is related to this Istio issue istio/istio#52208
Istio Ambient will create iptables rules that intercept kubelet readiness probes and set the source IP of the packet to 169.254.7.127
*nat
:ISTIO_POSTRT - [0:0]
-A POSTROUTING -j ISTIO_POSTRT
-A ISTIO_POSTRT -p tcp -m owner --socket-exists -m set --match-set istio-inpod-probes-v4 dst -j SNAT --to-source 169.254.7.127
The SYN packet is then routed to the pod with Cilium. The SYN-ACK return packet has a destination IP 169.254.7.127 and will be dropped by Cilium eBPF Routing and the return packet won't hit the iptables rules installed by Istio.
Calico eBPF had a similar bug here projectcalico/calico#9157 and they solved it here projectcalico/calico#9192
If Cilium BPF Routing can't route a packet that was coming from the host, it should return through host.
How can we reproduce the issue?
- Install Cilium with Helm
---
operator:
unmanagedPodWatcher:
restart: true
tunnelProtocol: geneve
cni:
customConf: false
uninstall: false
exclusive: false
ipam:
mode: cluster-pool
operator:
clusterPoolIPv4PodCIDRList:
- 10.245.0.0/16
directRoutingSkipUnreachable: true
policyEnforcementMode: default
bpf:
hostLegacyRouting: false
masquerade: true
socketLB:
hostNamespaceOnly: true
kubeProxyReplacement: true
k8sServiceHost: "k8s-master"
k8sServicePort: "6443"
egressGateway:
enabled: true
routingMode: native
autoDirectNodeRoutes: true
ipv4NativeRoutingCIDR: 10.245.0.0/16
- Install Istio Ambient
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
spec:
profile: ambient
- Create a deployment with http readinessProbe in a test namespace which has Istio Ambient enabled
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: web
name: web
namespace: test
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
name: http
readinessProbe:
httpGet:
path: /
port: http
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
Cilium Version
cilium-cli: v0.16.7 compiled with go1.22.2 on linux/amd64
cilium image (default): v1.15.4
cilium image (stable): v1.16.3
Kernel Version
6.8.0-45-generic
Kubernetes Version
v1.31.1
Regression
No response
Sysdump
No response
Relevant log output
Anything else?
No response
Cilium Users Document
- Are you a user of Cilium? Please add yourself to the Users doc
Code of Conduct
- I agree to follow this project's Code of Conduct