Temporal Scaler does not work properly with API Key authentication against Temporal Cloud as TLS is not enabled on the client: read: connection reset by peer #6703
Description
Report
I have a Temporal Cloud namespace using api_key authentication:
tcld n am get -n my-ns.snip
api_key
I tried to use the Temporal scaler and specified the apiKey using the TriggerAuthentication:
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: temporal-autoscaler-auth
namespace: default
spec:
secretTargetRef:
- parameter: apiKey
name: temporal-autoscaler-auth
key: apiKey
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: temporal-worker
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: temporal-worker
pollingInterval: 5
cooldownPeriod: 300
minReplicaCount: 5
maxReplicaCount: 15
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 15
triggers:
- type: cpu
metricType: Utilization
metadata:
value: "80"
- type: temporal
metadata:
namespace: my-ns.snip
taskQueue: main
targetQueueSize: 30
activationTargetValue: 1
activationTargetQueueSize: "0"
endpoint: "us-west1.gcp.api.temporal.io:7233"
maxConnectTimeout: "30"
authenticationRef:
name: temporal-autoscaler-authWhen applying this, the keda operator throws the error shown in the logs below ("read: connection reset by peer").
I believe this happens because the scaler does not set the TLS option when using just apiKey:
Lines 211 to 238 in 720ae0e
According to the docs, enabling TLS is required when using API Key: https://docs.temporal.io/cloud/api-keys#sdk
Expected Behavior
The Temporal scaler would work properly when connecting to Temporal Cloud with just an API Key provided
Actual Behavior
It gets a read: connection reset by peer network error.
Steps to Reproduce the Problem
- Create a Temporal Cloud namespace
- Configure an API key on it
- Try to use the Temporal trigger
Logs from KEDA operator
2025-04-09T14:37:30Z ERROR Error getting scalers {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-temporal-worker","namespace":"default"}, "namespace": "default", "name": "my-temporal-worker", "reconcileID": "94aeb370-a3fb-4979-8f4b-45c747166905", "error": "failed to create Temporal client connection: failed reaching server: connection error: desc = \"error reading server preface: read tcp 10.32.2.243:34834->34.82.7.179:7233: read: connection reset by peer\""}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
/workspace/controllers/keda/hpa.go:219
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
/workspace/controllers/keda/hpa.go:72
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
/workspace/controllers/keda/hpa.go:45
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
/workspace/controllers/keda/scaledobject_controller.go:452
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
/workspace/controllers/keda/scaledobject_controller.go:291
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
/workspace/controllers/keda/scaledobject_controller.go:193
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
KEDA Version
2.17.0
Kubernetes Version
Other
Platform
Google Cloud
Scaler Details
Temporal
Anything else?
Workaround:
Create a certificate and use it along side the API key, while mTLS is not enabled, this has the side-effect of enabling TLS, which will allow the Api Key authentication to work:
tcld gen ca --org my-org -d 1y --ca-cert ca.pem --ca-key ca.key
tcld gen leaf --org my-org -d 364d --ca-cert ca.pem --ca-key ca.key --cert client.pem --key client.key
base64 -w 0 client.pem
base64 -w 0 client.key
Then add these to the secret being used by the TriggerAuthentication and configure the TriggerAuthentication to also provide cert and key (while still providing the apiKey).
After this it works:
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/s1-temporal-my-ns-snip-main?labelSelector=scaledobject.keda.sh%2Fname%3Dtemporal-worker" -n keda{"kind":"ExternalMetricValueList","apiVersion":"external.metrics.k8s.io/v1beta1","metadata":{},"items":[{"metricName":"s1-temporal-my-ns-snip-main","metricLabels":null,"timestamp":"2025-04-09T15:04:33Z","value":"0"}]}