Unclosed socket warning after HTTP 407 response from HTTP CONNECT proxy #3374
Description
Subject
Running some HTTP proxy tests on our project. Getting unclosed socket warnings when an HTTP proxy returns with a 407 on CONNECT, causing our CI to fail. Digging a bit deeper it seems to be coming from urllib3.
It is only appearing on Python 3.10 and lower - I don't see this in 3.11 and 3.12.
Environment
Describe your environment.
At least, paste here the output of:
import platform
import ssl
import urllib3
print("OS", platform.platform())
print("Python", platform.python_version())
print(ssl.OPENSSL_VERSION)
print("urllib3", urllib3.__version__)OS Linux-6.8.1-1-default-x86_64-with-glibc2.39
Python 3.10.14
OpenSSL 3.1.4 24 Oct 2023
urllib3 2.2.1
Steps to Reproduce
Dummy server that returns with 407 for HTTP CONNECT:
from http.server import BaseHTTPRequestHandler
from socketserver import ThreadingTCPServer
class HTTPConnectProxyHandler(BaseHTTPRequestHandler):
protocol_version = 'HTTP/1.1'
default_request_version = 'HTTP/1.1'
def do_CONNECT(self):
self.send_response(407)
self.send_header('Proxy-Authenticate', 'Basic')
self.end_headers()
ThreadingTCPServer(('127.0.0.1', 55556), HTTPConnectProxyHandler).serve_forever()Urllib3 client to reproduce unclosed socket:
# test.py
import urllib3
with urllib3.ProxyManager(proxy_url=f'http://127.0.0.1:55556', retries=urllib3.util.retry.Retry(False)) as pm:
pm.urlopen('GET', 'https://example.com')We use urllib3.util.retry.Retry(False) to disable the retry logic in our project - we want the underlying proxy error.
- it still occurs without this
Expected Behavior
No unclosed socket warning
Actual Behavior
What happens instead.
You may attach logs, packet captures, etc.
$ python3.10 -Werror test.py
Traceback (most recent call last):
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 779, in urlopen
self._prepare_proxy(conn)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 1048, in _prepare_proxy
conn.connect()
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connection.py", line 633, in connect
self._tunnel() # type: ignore[attr-defined]
File "/usr/lib64/python3.10/http/client.py", line 925, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 407 Proxy Authentication Required
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/test.py", line 3, in <module>
pm.urlopen('GET', 'https://example.com')
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/poolmanager.py", line 634, in urlopen
return super().urlopen(method, url, redirect=redirect, **kw)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/poolmanager.py", line 444, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/util/retry.py", line 445, in increment
raise reraise(type(error), error, _stacktrace)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 779, in urlopen
self._prepare_proxy(conn)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 1048, in _prepare_proxy
conn.connect()
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connection.py", line 633, in connect
self._tunnel() # type: ignore[attr-defined]
File "/usr/lib64/python3.10/http/client.py", line 925, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
urllib3.exceptions.ProxyError: ('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))
Exception ignored in: <socket.socket [closed] fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
ResourceWarning: unclosed <socket.socket [closed] fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>If I remove the retries parameter to use the default we get even more unclosed socket warnings:
$ python3.10 -Werror test.py
Traceback (most recent call last):
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 779, in urlopen
self._prepare_proxy(conn)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 1048, in _prepare_proxy
conn.connect()
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connection.py", line 633, in connect
self._tunnel() # type: ignore[attr-defined]
File "/usr/lib64/python3.10/http/client.py", line 925, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 407 Proxy Authentication Required
The above exception was the direct cause of the following exception:
urllib3.exceptions.ProxyError: ('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/test.py", line 3, in <module>
pm.urlopen('GET', 'https://example.com')
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/poolmanager.py", line 634, in urlopen
return super().urlopen(method, url, redirect=redirect, **kw)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/poolmanager.py", line 444, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
return self.urlopen(
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
return self.urlopen(
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
return self.urlopen(
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/tmp/venv/lib64/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))
Exception ignored in: <socket.socket [closed] fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
ResourceWarning: unclosed <socket.socket [closed] fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
Exception ignored in: <socket.socket [closed] fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
ResourceWarning: unclosed <socket.socket [closed] fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
Exception ignored in: <socket.socket [closed] fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
ResourceWarning: unclosed <socket.socket [closed] fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
Exception ignored in: <socket.socket [closed] fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
ResourceWarning: unclosed <socket.socket [closed] fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>