Skip to content

Conversation

dadjeibaah
Copy link
Contributor

When Linkerd receives a significantly large HTTP request with a set content-length, the underlying Finagle router service buffers the entire message into memory instead of streaming the HTTP message in smaller chunks. This can cause problems when a client is sending very large file uploads without using Transfer-Encoding: Chunked

This PR pulls in a new Finagle param (#740) that forces a finagle server to receive large HTTP messages in smaller chunks when the HTTP messages content-length exceeds a certain content-length threshold. In addition, Linkerd exposes this Finagle param to allow for configurability. I was able to test this out in a docker environment where I sent a significantly large file (140MB) to Linkerd. Previously Linkerd would load buffer the entire message into memory and was easily visible in the JVM memory usage. After the change, the same request was issued again and we no longer observe the spike in memory.

fixes #2188

Signed-off-by: Dennis Adjei-Baah dennis@buoyant.io

Signed-off-by: Dennis Adjei-Baah <dennis@buoyant.io>
@dadjeibaah dadjeibaah self-assigned this Dec 11, 2018
@dadjeibaah dadjeibaah requested review from olix0r and adleong December 11, 2018 01:22
Copy link
Member

@adleong adleong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⭐ This looks good to me but it would be great to have someone test this against real production workloads, if possible. @edio: would you be interested in testing this out?

@koiuo
Copy link
Contributor

koiuo commented Dec 15, 2018

@dadjeibaah Thanks for this change, happy to see it in linkerd!

@adleong absolutely! I'm back on Tuesday however, if you think it is worth waiting till then I'd be happy to look at the change and give it a try. But I believe the only thing to discuss here is the default value of the parameter (I don't really like finagle's default 5 MiB), and it is not that important anyway.

Btw we just started using similar patch (based on linkerd 1.5.2) at NCBI.

According to my benchmarks the change also improves latency for fixed length requests. On my scenarios the difference is about 10-15% for requests below 1 MiB but it gets much more pronounced as request size grows larger or link to linkerd gets slower (I think everyone has linkerds on the same host with their apps, so it's all about copying data in memory anyway, hence only request size is a significant factor).

I have only single result on me right now, but if there's interest I could post more when I'm back

baseline
Client.invoke:invoke·p0.95               4096  sample           5.677           ms/op

vanilla l5d
Client.invoke:invoke·p0.95               4096  sample          10.879          ms/op

patched l5d
Client.invoke:invoke·p0.95               4096  sample           8.282          ms/op

Where baseline is single threaded direct invocation of a service on localhost which just counts bytes in request. Linkerd is executed on Java 8 with -Xms256m -Xmx256m. This case is 4MiB request.

@dadjeibaah dadjeibaah merged commit f7283a9 into master Dec 17, 2018
@dadjeibaah dadjeibaah deleted the dad/stream-large-requests-after branch December 17, 2018 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add new Finagle param FixedLengthStreamedAfter as a config option in LInkerd
3 participants