-
Notifications
You must be signed in to change notification settings - Fork 0
Add HTTPResponse.shutdown to stop blocking reads #27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
def shutdown(self) -> None: | ||
if not self._sock_shutdown: | ||
raise ValueError("Cannot shutdown socket as self._sock_shutdown is not set") | ||
self._sock_shutdown(socket.SHUT_RD) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shutdown
of pyOpenSSL sockets doesn't accept the parameter, we'll need to change its signature
urllib3/src/urllib3/contrib/pyopenssl.py
Lines 369 to 371 in b51fb3c
def shutdown(self) -> None: | |
# FIXME rethrow compatible exceptions should we ever use this | |
self.connection.shutdown() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great catch, fixed in 7a02982
(#27).
# Calling shutdown here calls shutdown() on the underlying socket, | ||
# so that the reamining read will fail instead of blocking | ||
# indefinitely | ||
response.shutdown() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling response.close()
here instead of response.shutdown()
doesn't make the read call hang
Would it need to block in case of calling close
for correct testing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The read() call does hang, actually.
The test itself does not hang, because after 5 seconds, the server will call sock.close()
itself. But in that case we don't get any data, so no ProtocolError
, simply the empty string, and the test will fail as expected.
I prefer failing after 5 seconds than hanging, because hanging tests are annoying to debug.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It didn't wait for five seconds with response.close()
, read returned right away, the following change printed 29980.729741648 and 29980.729781285
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
index 46feb4bf..ba45a2b5 100644
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -14,6 +14,7 @@ import socket
import ssl
import tempfile
import threading
+import time
import typing
import zlib
from collections import OrderedDict
@@ -1043,9 +1044,11 @@ class TestSocketClosing(SocketDummyServerTestCase):
# Calling shutdown here calls shutdown() on the underlying socket,
# so that the remaining read will fail instead of blocking
# indefinitely
- response.shutdown()
- with pytest.raises(ProtocolError, match="Connection broken"):
- response.read()
+ response.close()
+ # with pytest.raises(ProtocolError, match="Connection broken"):
+ print(time.monotonic())
+ response.read()
+ print(time.monotonic())
timed_out.set()
No protocol error raised indeed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, the test as a whole takes 5 seconds, but only because the teardown only completes when socket_handler
returns. I initially had two client threads to recreate the original report, and they did not seem to be needed.
They are in fact needed, so I'll try to add that back to reproduce the initial report better, thanks.
It could cause a memory leak.
* Upgrade pyupgrade to 3.19.0 * Upgrade black to 24.10.0 * Upgrade isort to 5.13.2 * Upgrade flake8 to 7.1.1 --------- Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>
Co-authored-by: Illia Volochii <illia.volochii@gmail.com>
No description provided.