Skip to content

Default chart values break outbound HTTPS in Istio clusters #500

@jhuffman-wyn

Description

@jhuffman-wyn

Discovered a weird one today. If you have the following setup:

  1. Keda installed with admission webhook (default settings)
  2. Istio installed with applications using Istio sidecar containers

...all outbound HTTPS traffic will be blocked with mysterious "SSL protocol failure" and similar errors. This is due to an unpleasant interaction between keda and Istio, and while the onus may be on Istio to fix, keda can also proactively avoid contributing to it.

Expected Behavior

HTTPS traffic is able to reach external endpoints without issue.

Actual Behavior

All HTTPS traffic on pods with Istio sidecars is blocked with TLS/SSL protocol errors.

Steps to Reproduce the Problem

  1. Install Istio (any recent version) with default settings.
  2. Install current keda version with default settings.
  3. Have at least one pod with an Istio sidecar.
  4. From within that pod, attempt to reach any external HTTPS endpoint.
  5. You will get a TLS/SSL error.

Note that this blog is what informed me of the issue: https://fable.sh/blog/istio-port-443-and-ssl-errors./

That's not specific to keda, but it is apparent that the Istio fix did not account for every scenario: istio/istio#16458

This is being logged as a keda bug as changing the admission webhook service can easily avoid causing this problem:

apiVersion: v1
kind: Service
metadata:
  name: keda-admission-webhooks
spec:
  ...
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 443 
    protocol: TCP
    targetPort: 9443
  selector:
    app: keda-admission-webhooks
  sessionAffinity: None
  type: ClusterIP

The specific combination of the port name being http and the port being 443 causes this. Changing either of those values will avoid this. Calling it http-web, for instance, will do the job just fine.

Specifications

  • KEDA Version: 2.11.1
  • Platform & Version:
  • Kubernetes Version: GKE 1.24
  • Scaler(s): N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions