Skip to content

Cilium is confused about source address if io.cilium.proxy-visibility observes http traffic #25630

@farcaller

Description

@farcaller

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Here's the setup:

I have two clusters A and B with cilium on both running in cluster mesh (native routing with kube-router doing the BGP only).

A pod ingress in cluster A receives http traffic and forwards it to a pod workload in cluster B. Workload is annotated with io.cilium.proxy-visibility: '<Egress/53/UDP/DNS>,<Egress/80/TCP/HTTP>,<Ingress/80/TCP/HTTP>'.

(coincidentally, ingress is an envoy instance that's configured by istio. Istio service mesh isn't used and istio is pretty much a thin k8s CRD wrapper on top of envoy).

When the abovementioned annotation is present and I curl to ingress, the hubble reports that the traffic to workload on B originated from 'world' and shows the IP address of the machine with curl. That's seems to be wrong, because from the cluster perspective the traffic originated from the cluster A pod ingress. I suspect that cilium's envoy on B that gets enabled due to proxy-visibility trusts the ingress'es x-forwarded-for for whatever reason.

This makes it impossible to set up the reasonable CiliumNetowrkPolicies, because I want workload to receive traffic from ingress only, but the flows are reported from the out-of-cluster machines.

Cilium Version

1.13.2

Kernel Version

Linux web 6.1.27 #1-NixOS SMP PREEMPT_DYNAMIC Sun Apr 30 23:26:28 UTC 2023 x86_64 GNU/Linux

Kubernetes Version

v1.26.4+k3s1

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/agentCilium agent related.area/clustermeshRelates to multi-cluster routing functionality in Cilium.area/proxyImpacts proxy components, including DNS, Kafka, Envoy and/or XDS servers.area/servicemeshGH issues or PRs regarding servicemeshkind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions