CiliumLocalRedirectPolicy skipRedirectFromBackend does not work as expected

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Version

equal or higher than v1.17.0 and lower than v1.18.0

### What happened?

We are attempting to configure an nginx proxy in front of AWS IMDS service on "169.254.169.254" so we can allowlist only the endpoints we want accessible to pods in the cluster. When we configure a CiliumLocalRedirectPolicy loosly following the kiam example, https://docs.cilium.io/en/stable/network/kubernetes/local-redirect-policy/#kiam-redirect-on-eks, if `skipRedirectFromBackend` is set to true, pod traffic is not hijacked. If we set `skipRedirectFromBackend` to `false` all pod traffic is hijacked as expected, but that includes the nginx proxy pod, which results in an infinite loopback when talking to the proxy.

So it seems `skipRedirectFromBackend` does not work as expected, or at all.

We initially tested on Cilium v1.16.2, but upgraded our testing cluster to Cilium 1.17.0-rc.0, but it did not change the behavior. System Info:
```
  Kernel Version:                             6.1.119-129.201.amzn2023.x86_64
  OS Image:                                   Amazon Linux 2023.6.20241212
  Operating System:                           linux
  Architecture:                               amd64
  Container Runtime Version:                  containerd://1.7.23
  Kubelet Version:                            v1.30.7-eks-59bf375
  Kube-Proxy Version:                         v1.30.7-eks-59bf375
```

### How can we reproduce the issue?

1. Cilium installed w/helm (via ArgoCD) in an EKS cluster. Values:
```
cilium:
  extraEnv:
    - name: MALLOC_ARENA_MAX
      value: "1"
  k8sServicePort: 443
  localRedirectPolicy: true
  egressMasqueradeInterfaces: ens+
  extraArgs:
    - "--api-rate-limit"
    - "endpoint-create=rate-limit:5/s,rate-burst:40,parallel-requests:40" # 10x base rate limits, see https://docs.cilium.io/en/stable/configuration/api-rate-limiting/
  
  serviceAccounts:
    nodeinit:
      enabled: true
  
  nodeinit:
    enabled: true
    priorityClassName: system-node-critical
    podAnnotations:
      security.apps.indeed.com/podsecurityexception: "cilium"
      reloader.stakater.com/auto: "true"
  cni:
    # Defaults to true.  Setting to false avoids removal of CNI configuration
    # files during upgrades in order to ensure nodes do not go unmanageable.
    uninstall: false
  operator:
    priorityClassName: system-cluster-critical
    podAnnotations:
      reloader.stakater.com/auto: "true"
      security.apps.indeed.com/podsecurityexception: "cilium"
      ad.datadoghq.com/tolerate-unready: "true"
      ad.datadoghq.com/cilium-operator.checks: |
        {
          "openmetrics": {
            "instances": [
              {
                "openmetrics_endpoint": "http://%%host%%:9963/metrics",
                "metrics": [
                  {"cilium_operator_ipam_available_ips_per_subnet": "cilium.operator.ipam.available.ips_per_subnet"},
                  {"cilium_operator_ipam_available_ips": "cilium.operator.ipam.available.ips"},
                  {"cilium_operator_ipam_ips": "cilium.operator.ipam.ips"},
                  {"cilium_operator_process_resident_memory_bytes": "cilium.operator.process.resident_memory.bytes"}
                ]
              }
            ]
          }
        }
    podDisruptionBudget:
      enabled: true
    prometheus:
      enabled: true

  # cilium agent
  podAnnotations:
    reloader.stakater.com/auto: "true"
    security.apps.indeed.com/podsecurityexception: "cilium"
    ad.datadoghq.com/tolerate-unready: "true"
    ad.datadoghq.com/cilium-agent.logs: >-
      [{
        "source": "cilium-agent",
        "service": "cilium-agent",
        "log_processing_rules": [{
          "type": "include_at_match",
          "name": "include_hubble_flow",
          "pattern" : "^\\{.*namespace.*\\}$"
        }]
      }]
    ad.datadoghq.com/cilium-agent.checks: |
      {
        "openmetrics": {
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:9962/metrics",
              "metrics": [
                {"cilium_api_limiter_processed_requests_total": "cilium.api_limiter.processed_requests"},
                {"cilium_api_limiter_processing_duration_seconds": "cilium.api_limiter.processing_duration.seconds"},
                {"cilium_api_limiter_requests_in_flight": "cilium.api_limiter.requests_in_flight"},
                {"cilium_bpf_map_ops_total": "cilium.bpf.map_ops"},
                {"cilium_bpf_map_pressure": "cilium.bpf.map_pressure"},
                {"cilium_drop_count_total": "cilium.drop_count"},
                {"cilium_endpoint_regenerations_total": "cilium.endpoint.regenerations"},
                {"cilium_endpoint_state": "cilium.endpoint.state"},
                {"cilium_forward_count_total": "cilium.forward_count"},
                {"cilium_identity": "cilium.identity"},
                {"cilium_ip_addresses": "cilium.ip_addresses"},
                {"cilium_ipam_events_total": "cilium.ipam.events"},
                {"cilium_k8s_client_api_calls_total": "cilium.k8s_client.api_calls"},
                {"cilium_k8s_client_api_latency_time_seconds": "cilium.k8s_client.api_latency_time.seconds"},
                {"cilium_policy": "cilium.policy"},
                {"cilium_policy_endpoint_enforcement_status": "cilium.policy.endpoint_enforcement_status"},
                {"cilium_endpoint_propagation_delay_seconds": "cilium.endpoint.propagation_delay.seconds"},
                {"cilium_endpoint_regeneration_time_stats_seconds": "cilium.endpoint.regeneration.seconds"},
                {"cilium_process_cpu_seconds_total": "cilium.process.cpu.seconds"},
                {"cilium_process_resident_memory_bytes": "cilium.process.resident_memory.bytes"},
                {"cilium_unreachable_nodes": "cilium.unreachable.nodes"},
                {"cilium_unreachable_health_endpoints": "cilium.unreachable.health_endpoints"}
              ]
            }
          ]
        }
      }

  kubeProxyReplacement: true

  prometheus:
    enabled: true

  socketLB:
    enabled: true

  # workaround for cilium mem leak in l2 neighbor discovery error handling in 1.16.0-1.16.1+
  l2NeighDiscovery:
    enabled: false

  ipam:
    mode: eni
  routingMode: native
    ciliumNodeUpdateRate: 5s # Ref https://github.com/cilium/cilium/pull/23017
    operator:
      externalAPILimitBurstSize: 100
      externalAPILimitQPS: 10.0
  k8sClientRateLimit:
    burst: 100
    qps: 20
  ciliumEndpointSlice:
    enabled: true
    rateLimits:
    - nodes: 0
      limit: 10
      burst: 20
    - nodes: 50
      limit: 20
      burst: 50
    - nodes: 100
      limit: 25
      burst: 100
    - nodes: 500
      limit: 50
      burst: 100
  # ignore any of these labels for identity purposes, means they can not be used in network policy pod selectors
  # in general we want to ignore any label that forces uniqueness per pod
  # Ref https://docs.cilium.io/en/latest/operations/performance/scalability/identity-relevant-labels/#identity-relevant-labels
  # labels are regex prefixes, if both inclusive and exclusive match, the longer match wins
  labels: >-
    k8s:kubernetes\\.io/metadata\\.name$
    k8s:app\\.kubernetes\\.io/component$
    k8s:app\\.kubernetes\\.io/instance$
    k8s:app\\.kubernetes\\.io/name$
    k8s:apps\\.indeed\\.com/project$
    k8s:apps\\.indeed\\.com/product-group$
    k8s:apps\\.indeed\\.com/env$
    k8s:apps\\.indeed\\.com/deployment-group-name$
    k8s:apps\\.indeed\\.com/mesh-service-name$
    k8s:io\\.kubernetes\\.pod\\.namespace$
    k8s:io\\.cilium\\.k8s\\.policy\\.serviceaccount$
    k8s:app$
    k8s:apps$
    k8s:cluster$
    k8s:component$
    k8s:env$
    k8s:hub\\.jupyter\\.org/network-access-hub$
    k8s:hub\\.jupyter\\.org/network-access-proxy-api$
    k8s:hub\\.jupyter\\.org/network-access-proxy-http$
    k8s:hub\\.jupyter\\.org/network-access-singleuser$
    k8s:name$
    k8s:release$
    k8s:role$

  bpf:
    policyMapMax: 65536 # limited to 16 bits https://github.com/cilium/cilium/issues/27866
    mapDynamicSizeRatio: 0.005 # Ref https://docs.cilium.io/en/stable/network/ebpf/maps/

  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 10%

  # Starting in cilium v1.16, Cilium Envoy is enabled by default. This is a
  # DaemonSet that is required only for L7 Policies/Traffic Inspection
  envoy:
    enabled: true
    priorityClassName: system-node-critical
    podAnnotations:
      security.apps.indeed.com/podsecurityexception: "cilium"
  loadBalancer:
    l7:
      backend: envoy

  hubble:
    peerService:
      # Note the final `.` to make this fully-qualified
      clusterDomain: cluster.local.
    relay:
    # The Hubble Relay is a stand-alone component that collects network flow
    # data from each cilium instance and makes it available to the Hubble UI
    # and CLI via a set of APIs.
      enabled: true
      replicas: 2
      rollOutPods: true
      resources:
        requests:
          cpu: 100m
          memory: 100Mi
      priorityClassName: observability-critical
      podAnnotations:
        reloader.stakater.com/auto: "true"
      podDisruptionBudget:
        enabled: true
        maxUnavailable: 1
    redact:
      enabled: true
      http:
        userInfo: true
    ui:
      enabled: true
      # We can't run more than 1 UI replicas until this issue is fixed
      # https://github.com/cilium/hubble-ui/issues/833
      replicas: 1
      rollOutPods: true
      backend:
        # Keeping probes disabled until this issue is fixed https://github.com/cilium/hubble-ui/issues/853
        livenessProbe:
          enabled: false
        readinessProbe:
          enabled: false
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
      frontend:
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
      priorityClassName: observability-critical
      podAnnotations:
        reloader.stakater.com/auto: "true"
    metrics:
      enabled:
        - dns:query;ignoreAAAA
        - drop
        - tcp
        - flow
        - icmp
        - http
        - policy:sourceContext=app|workload-name|pod|reserved-identity;destinationContext=app|workload-name|pod|dns|reserved-identity;labelsContext=source_namespace,destination_namespace
        - flow:sourceContext=workload-name|reserved-identity;destinationContext=workload-name|reserved-identity
      enableOpenMetrics: true
    tls:
      auto:
        method: certmanager
        certManagerIssuerRef:
          group: cert-manager.io
          kind: Issuer
          name: hubble-ca-issuer
    export:
      static:
        enabled: true
        filePath: stdout
        fieldMask:
        - time
        - source.namespace
        - source.pod_name
        - source.identity
        - IP.source
        - IP.destination
        allowList:
        - '{"destination_ip":["169.254.169.254"]}'
```

2. Deploy the following to set up the imds-proxy and local redirect:
[deployment.zip](https://github.com/user-attachments/files/18204118/deployment.zip)

3. Start up a ubuntu with a shell, run:
```
apt update && apt install curl -y
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/
```

Expect to see the request hijacked and denied by the imds-proxy service. However the request succeeds and goes to the world. Hubble shows the request is never hijacked. Note, if we remove `skipRedirectFromBackend` from the local redirect policy, the requests are hijacked as expected.

### Cilium Version

1.17.0.rc.0

### Kernel Version

`6.1.119-129.201.amzn2023.x86_64`

### Kubernetes Version

`Server Version: v1.30.6-eks-7f9249a`

### Regression

_No response_

### Sysdump

_No response_

### Relevant log output

```shell

```

### Anything else?


### Cilium Users Document

- [ ] Are you a user of Cilium? Please add yourself to the [Users doc](https://github.com/cilium/cilium/blob/main/USERS.md)

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CiliumLocalRedirectPolicy skipRedirectFromBackend does not work as expected #36740

Is there an existing issue for this?

Version

What happened?

How can we reproduce the issue?

Cilium Version

Kernel Version

Kubernetes Version

Regression

Sysdump

Relevant log output

Anything else?

Cilium Users Document

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CiliumLocalRedirectPolicy skipRedirectFromBackend does not work as expected #36740

Description

Is there an existing issue for this?

Version

What happened?

How can we reproduce the issue?

Cilium Version

Kernel Version

Kubernetes Version

Regression

Sysdump

Relevant log output

Anything else?

Cilium Users Document

Code of Conduct

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions