http2 requests fail with "protocol error: The user callback function failed" #6421
Description
Describe the bug
Using the daily-build/master-20180615-09-15, http2 requests to an HTTPS service fail in the sidecar with the following message:
[2018-06-20 13:37:29.830][30][info][client] external/envoy/source/common/http/codec_client.cc:117] [C6] protocol error: The user callback function failed
[2018-06-20T13:37:29.825Z] "GET / HTTP/2" 503 UC 0 57 5 - "10.132.0.8" "curl/7.60.0" "5c2bfba2-ea0d-932b-b937-dbe03cdda2df" "httpbin.mydomain.xyz" "127.0.0.1:80"
http1.1 requests are successful:
$ curl -I --http1.1 https://httpbin.mydomain.xyz
HTTP/1.1 200 OK
server: envoy
date: Wed, 20 Jun 2018 13:41:01 GMT
content-type: text/html; charset=utf-8
content-length: 7814
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 111
But http2 requests fail:
$ curl -I --http2 https://httpbin.mydomain.xyz
HTTP/2 503
content-length: 57
content-type: text/plain
date: Wed, 20 Jun 2018 13:46:03 GMT
server: envoy
x-envoy-upstream-service-time: 88
Expected behavior
http2 requests should succeed.
Steps to reproduce the bug
Deploy Istio into a fresh GKE 1.9.7-gke.3 cluster, with cert-manager installed and a certificate created for the *.mydomain.xyz domain.
Edit the istio config map to add istio-system namespace to the statsdUdpAddress: statsdUdpAddress: istio-statsd-prom-bridge.istio-system:9125
$ helm install install/kubernetes/helm/istio --name istio --namespace istio-system \
--set global.mtls.enabled=true \
--set global.controlPlaneSecurityEnabled=true \
--set global.proxy.includeIPRanges="10.28.0.0/14\,10.31.240.0/20" \
--set global.proxy.resources.limits.cpu=100m \
--set global.proxy.resources.limits.memory=128Mi \
--set ingressgateway.enabled=true \
--set ingress.enabled=false \
--set egressgateway.enabled=false \
--set prometheus.enabled=false
Deploy httpbin service. Update the gateway host to something available.
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Namespace
metadata:
name: httpbin
labels:
name: httpbin
istio-injection: enabled
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: gateway
namespace: httpbin
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- '*.mydomain.xyz'
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
namespace: httpbin
spec:
hosts:
- httpbin.mydomain.xyz
gateways:
- gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
host: httpbin.httpbin.svc.cluster.local
---
apiVersion: v1
kind: Service
metadata:
name: httpbin
namespace: httpbin
labels:
app: httpbin
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: httpbin
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: httpbin
namespace: httpbin
spec:
replicas: 1
template:
metadata:
labels:
app: httpbin
version: v1
spec:
containers:
- image: kennethreitz/httpbin
imagePullPolicy: Always
name: httpbin
ports:
- containerPort: 80
EOF
Version
What version of istio and Kubernetes are you using? Use istioctl version and kubectl version
Istio release https://gcsweb.istio.io/gcs/istio-prerelease/daily-build/master-20180615-09-15/
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.3", GitCommit:"9b5b719c5f295c99de68ffb5b63101b0e0175376", GitTreeState:"clean", BuildDate:"2018-05-31T18:32:23Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
Is Istio Auth enabled or not?
Enabled, command line above.
Environment
GKE