Title:
When making client requests that contain invalid chars in an HTTP header to an Envoy instance ("envoy-one") that's configured to proxy over H/2 to an upstream Envoy ("envoy-two"), we see envoy-one return a 503 response with response flags "UR", without proxying the request to “envoy-two”. If we change the Envoy ←→ Envoy communication to use HTTP/1.1, we don’t see this behavior, and the request is happily proxied to the upstream Envoy.

Description:
While investigating some strange errors we were seeing in one of our applications, I wrote a quick test harness, so that I could make requests that contained invalid ASCII chars in HTTP headers.

The “invalid ASCII chars” are defined by the relevant RFC specs [0] [1]. In this specific test case, I inserted an ASCII control char (ASCII value 31, or 0x1F) into an HTTP header value and executed that request against our stack (Envoy → Envoy → application).

Relevant Links:
[0] https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2.2

I saw that this would generate a 503 from the first Envoy before the request ever reaches our application, since it sits upstream of the second Envoy.

Repro steps:
The repro config and necessary files are located in a github repo: https://github.com/dcarney/envoy-repro

Reproing this scenario involves setting up the following topography:

I have the HTTP request saved in the request.txt file. It includes the HTTP header value with an invalid ASCII control character, along with other random characters.

The included Docker Compose file sets up the following:

• envoy-one
  • a listener on port :10001
  • a cluster configured to use HTTP/2
  • all traffic from the listener is sent to cluster “envoy-two” on port :10002
• envoy-two
  • a listener on port :10002
  • a cluster configured to point to https://www.google.com (The actual upstream is irrelevant, as we’ll see)

Each envoy has its listener port and admin port mounted to the host, and the two Envoys are able to reach each other over the Docker network “envoytest”.

Build the two Envoy containers, and start everything up with:

$ docker build -t envoy:one -f Dockerfile.one .
$ docker build -t envoy:two -f Dockerfile.two .
$ docker-compose up

In another terminal, run the following, in order to make the HTTP request to the first Envoy (“envoy-one”):

$ cat request.txt | nc -n -i 127.0.0.1:10001

You should see the 503 response, which is emitted by the first Envoy (“envoy-one”):

HTTP/1.1 503 Service Unavailable
content-length: 85
content-type: text/plain
date: Wed, 22 May 2019 20:43:43 GMT
server: envoy

upstream connect error or disconnect/reset before headers. reset reason: remote reset%

Envoy access logs for the “client Envoy” (envoy-one) show that the “response flags” for this request are “UR”, which according to the Envoy docs means:

UR: Upstream remote reset in addition to 503 response code.

We can change the connection between “envoy-one” and “envoy-two” to use HTTP/1.1, by removing line 43-45 in envoy-one.yaml:

35   clusters:
...
40     dns_lookup_family: V4_ONLY
41     lb_policy: ROUND_ROBIN
42     tls_context:
43     http2_protocol_options: 
44       hpack_table_size:
45       max_concurrent_streams:
46     load_assignment:
47       cluster_name: service_foobar

If we re-build the containers, restart the Docker Compose topology, and execute the request again, we find that we now get a 404 response from https://www.google.com/foo, which is what we expect when the request is successfully proxied through both Envoys.

I've included packet captures from tcpdump from both Envoys in the repo (envoy-{one,two}.pcap). Looking at them in Wireshark clearly shows the upstream Envoy resetting the stream because of a "protocol error":

Similarly, in the trace logs, we see this message from the "client Envoy":

envoy-one_1 [2019-05-23 16:31:36.076][11][debug][router] [source/common/router/router.cc:676] [C0][S8542315413287350126] upstream reset: reset reason remote reset

And this message from the upstream Envoy:

envoy-two_1 [2019-05-23 16:31:36.074][11][debug][http2] [source/common/http/http2/codec_impl.cc:539] [C0] invalid frame: Invalid HTTP header field was received on stream 1

I would expect that the client Envoy would return a 400 error, rather than constructing an H/2 request that fails to parse in the upstream Envoy.

Config:
envoy-one config
envoy-two config

Logs:
trace logs from both Envoys

I'm happy to provide any more details you might need. Thanks for looking!