[WebServerPlugin ] Add support for Asynchronous response streaming #289
Description
Describe the bug
WebServerPlugin does not render all data from upstream in a timely manner
To Reproduce
Steps to reproduce the behavior:
- Run
proxy.pyas '--enable-webserver --plugin proxy.plugin.WebServerPlugin' - Some modifications to the WebServerPlugin have been done as explained in the context section
- Do a curl to trigger error
- See error
Expected behavior
All response data from upstream should come through in a timely fashion. But instead the socket is held for sometime by the proxy/plugin even after reading upstream data and there is a delay in sending the data back to the client
Version information
- OS: Mac OS Caltalina
- Browser curl
- Device: Mac
- proxy.py Version [e.g. 1.1.1]
Additional context
Following changes have been done to the WebServerPlugin. Note that the route has a * which is supposed to match all.
The upstream server is hosted at local port 5678 and hosts a plain css file.
The modifications are inspired by the plugin reverse_proxy to reach to some upstream server but note that these modifications are required for this plugin to work correctly to process all data. The original code in the reverse_proxy plugin just does a single conn.recv call which is not enough to handle larger datasets.
class WebServerPlugin(HttpWebServerBasePlugin):
"""Demonstrates inbuilt web server routing using plugin."""
def routes(self) -> List[Tuple[int, str]]:
return [
(httpProtocolTypes.HTTP, r'/*'),
]
def handle_request(self, request: HttpParser) -> None:
upstream = b'http://localhost:5678'
url = urlparse.urlsplit(upstream)
assert url.hostname
with socket_connection((text_(url.hostname), url.port if url.port else DEFAULT_HTTP_PORT)) as conn:
while True:
print("I am reading data")
data = conn.recv(DEFAULT_BUFFER_SIZE)
print(len(data))
self.client.queue(memoryview(data))
print("pushed data")
if not data:
print("breaking from loop")
break
Now run the following curl command. Note the the css file is just a random css file of size 268K
curl -v http://localhost:8899/css/app.cff42e0c.cs
The output of the curl command is the following. Note that the transfer took upwards of 5 seconds whereas the code has no such delay or sleep. By running the code with the debug prints, it becomes evident that data is read from the upstream socket very quickly, but it is not returned to the client for sometime.
curl -v http://localhost:8899/css/app.cff42e0c.css >o
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
> GET /css/app.cff42e0c.css HTTP/1.1
> Host: localhost:8899
> User-Agent: curl/7.64.1
> Accept: */*
>
0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: http://localhost:5678
< Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, PATCH, DELETE
< Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, sessionid
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Last-Modified: Sat, 01 Feb 2020 23:47:30 GMT
< ETag: W/"43093-1700326f6c3"
< Content-Type: text/css; charset=UTF-8
< Content-Length: 274579
< Vary: Accept-Encoding
< Date: Tue, 04 Feb 2020 21:04:27 GMT
< Connection: keep-alive
<
{ [54576 bytes data]
100 268k 100 268k 0 0 54697 0 0:00:05 0:00:05 --:--:-- 72030
* Connection #0 to host localhost left intact
* Closing connection 0