@soya3129 could you clarify to @tatsuhiro-t how this helps downstream Envoy metadata work more smoothly?

Also @tatsuhiro-t if there's anything you'd like to see here in the way of testing, please let us know!

The change will help Envoy to redirect requests. In regular request handling, when the upstream connection is ready to receive a request, envoy always submits headers first to nghttp2, and the submitted headers are then immediately drained by calling nghttp2_session_send().

Unlike the regular case where headers submitting happens when an upstream connection is available, the redirecting happens when nghttp2 is in the middle of decoding the response. When envoy knows redirecting is needed, it tries to recreate the same request by submitting headers to nghttp2. But envoy will not call nghttp2_session_send() to drain the headers frames until it returns from response processing. When envoy recreates the request, some other frame types may also get submitted to nghttp2, for example, extension frames which end up in session->ob_reg. Because session->ob_reg is popped before session->ob_syn (headers), extension frames is handled before a stream is created.

The problem can be handled in envoy, if envoy only submits headers for redirected requests until nghttp2_session_send() is called. But it would be much simpler to implement if envoy submits all the frames available when recreating the redirecting request. I think it might also OK for nghttp2 to process session->ob_syn before session->ob_reg. But I know so little about nghttp2, and I could be totally wrong, :)

+1 to alyssawilk@'s point. Please let me know what test case you would like to see to make the change. Thanks a lot!!

cc @birenroy

I tend to disagree with this change.
While this change might help the particular use case for envoy, other applications might want the current ordering of frames. In general, I think application should not depend on the intimate knowledge of the ordering of frames libnghttp2 emits.

Hi Tatsuhiro, I don't understand your objection. The HTTP/2 specification itself says that frame order is important.

https://httpwg.org/specs/rfc7540.html#StreamsLayer
"The order in which frames are sent on a stream is significant. Recipients process frames in the order they are received. In particular, the order of HEADERS and DATA frames is semantically significant."

Since HEADERS are used to open streams, shouldn't HEADERS frames be flushed first, if they are available? In particular, if a user of the library queues a HEADERS frame for stream k and another non-DATA frame for stream k, the current ordering would result in a well-behaved peer returning a PROTOCOL_ERROR.

It says nothing about extension frames. There is a frame which can be sent before HEADERS (e.g., PRIORITY).