Subject

Originally reported in boto/botocore#3111, we noticed users started experiencing TypeErrors being raised from urllib3 around the time 2.2.0 was released. After a deeper dive, it appears a change was made in 4ece59b that started indiscriminately casting headers passed to urlopen as an HTTPHeaderDict. This has specific implications for header values that are defined as bytes, while this doesn't match the current typing information, urllib3 (and Requests) both supported this prior to the release of urllib3 2.0.

Talking to @pquentin, it sounds like this is still an intended use case, but there isn't a formal contract at this time. Below I've included a minimal example to demonstrate the error. We would like to get confirmation if this is an undocumented breakage in 2.x or if this is unintended behavior. From the perspective of HTTPHeaderDict in isolation, it seems odd to allow instantiation when it fails immediately on use.

I'm happy to look at supplying a patch once we determine the intended behavior.

Environment

Reproducible on all platforms with urllib3 2.2.0.

Steps to Reproduce

from urllib3._collections import HTTPHeaderDict

hdict = HTTPHeaderDict({'x-my-header': b'my-binary-value'})
# Any access that invokes `.get()` will fail (print, get, getall, etc.)
print(hdict)

Expected Behavior

HTTPHeaderDict either fails at initialization or supports bytes values.

Actual Behavior

>>> print(hdict)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/.venv/lib/python3.11/site-packages/urllib3/_collections.py", line 423, in __repr__
    return f"{type(self).__name__}({dict(self.itermerged())})"
                                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv/lib/python3.11/site-packages/urllib3/_collections.py", line 446, in itermerged
    yield val[0], ", ".join(val[1:])
                  ^^^^^^^^^^^^^^^^^^