Skip to content

Add support for Python 3.12 #2188

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 19, 2025
Merged

Conversation

sirosen
Copy link
Member

@sirosen sirosen commented Jun 13, 2025

  • Add 3.12 to CI
  • Update a test which fails on 3.12 due to changes in CPython/pip behaviors, with a comment explaining why
Contributor checklist
  • Included tests for the changes.
  • PR title is short, clear, and ready to be included in the user-facing changelog.
Maintainer checklist
  • Verified one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
  • Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

sirosen and others added 3 commits May 22, 2025 11:01
`click` v8.2 needs to be handled -- as a follow-up change -- but it's
hard to know if the application itself will work with it yet. Instead
of restricting that in tox.ini, I have chosen to restrict it in the
metadata itself.

`setuptools` on newer versions breaks some of the tests, but
restricting it for package metadata is definitely wrong. Therefore, it
is only restricted for now on the `pipsupported` build.

This should lead to improved CI, but will not result in all tests
passing.
Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
The stderr may stop after noting the missing version,
without mentioning the missing name
@sirosen
Copy link
Member Author

sirosen commented Jun 13, 2025

I didn't think to check 3.12-lowest locally before putting up this PR. 🤔 I bet I can fix that, but I'll need to work out how.

@sirosen sirosen force-pushed the support-python312 branch from ec8e0ba to 8ff350b Compare June 13, 2025 21:14
@sirosen
Copy link
Member Author

sirosen commented Jun 14, 2025

Both of the Windows builds are failing on the same test.

raw error log from pytest
________________ test_invalid_pip_version_in_python_executable ________________
[gw2] win32 -- Python 3.12.10 C:\a\pip-tools\pip-tools\.tox\pipsupported-coverage\Scripts\python.EXE

get_pip_version_for_python_executable = <MagicMock name='get_pip_version_for_python_executable' id='2746640405168'>
runner = <click.testing.CliRunner object at 0x0000027F808F91C0>
tmp_path = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-unknown/pytest-0/popen-gw2/test_invalid_pip_version_in_py0')

    @mock.patch("piptools.scripts.sync.get_pip_version_for_python_executable")
    def test_invalid_pip_version_in_python_executable(
        get_pip_version_for_python_executable, runner, tmp_path
    ):
        with open(sync.DEFAULT_REQUIREMENTS_FILE, "w") as req_in:
            req_in.write("small-fake-a==1.10.0")
    
        custom_executable = tmp_path / "custom_executable"
        custom_executable.write_text("")
    
        custom_executable.chmod(0o700)
    
        get_pip_version_for_python_executable.return_value = Version("19.1")
    
        out = runner.invoke(cli, ["--python-executable", str(custom_executable)])
        assert out.exit_code == 2, out
        message = (
            "Target python executable '{}' has pip version 19.1 installed. "
            "Version"  # ">=20.3 is expected.\n" part is omitted
        )
>       assert out.stderr.startswith(message.format(custom_executable))
E       assert False
E        +  where False = <built-in method startswith of str object at 0x0000027F807B7C90>("Target python executable 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-unknown\\pytest-0\\popen-gw2\\test_invalid_pip_version_in_py0\\custom_executable' has pip version 19.1 installed. Version")
E        +    where <built-in method startswith of str object at 0x0000027F807B7C90> = "Could not resolve 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-unknown\\pytest-0\\popen-gw2\\test_invalid_pip_version_in_py0\\custom_executable' as valid executable path or alias.\n".startswith
E        +      where "Could not resolve 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-unknown\\pytest-0\\popen-gw2\\test_invalid_pip_version_in_py0\\custom_executable' as valid executable path or alias.\n" = <Result SystemExit(2)>.stderr
E        +    and   "Target python executable 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-unknown\\pytest-0\\popen-gw2\\test_invalid_pip_version_in_py0\\custom_executable' has pip version 19.1 installed. Version" = <built-in method format of str object at 0x0000027F80541DF0>(WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-unknown/pytest-0/popen-gw2/test_invalid_pip_version_in_py0/custom_executable'))
E        +      where <built-in method format of str object at 0x0000027F80541DF0> = "Target python executable '{}' has pip version 19.1 installed. Version".format

Looks like it's a test which creates a dummy/stub file to use as a fake python executable, and then passes that to --python-executable for the test.
The failure, dug out of that log, is

Could not resolve 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-unknown\\pytest-0\\popen-gw2\\test_invalid_pip_version_in_py0\\custom_executable' as valid executable path or alias.\n

That comes from scripts/sync.py, where there's this check:

resolved_python_executable = shutil.which(python_executable)
if resolved_python_executable is None:
    msg = "Could not resolve '{}' as valid executable path or alias."
    log.error(msg.format(python_executable))
    sys.exit(2)

shutil.which has notes about Python 3.12 changes on Windows, specifically. Current doc:

Changed in version 3.12: On Windows, the current directory is no longer prepended to the search path if mode includes os.X_OK and WinAPI NeedCurrentDirectoryForExePathW(cmd) is false, else the current directory is prepended even if it is already in the search path; PATHEXT is used now even when cmd includes a directory component or ends with an extension that is in PATHEXT; and filenames that have no extension can now be found.

That phrasing sounds like nothing should be impactful here, but my speculation is that the updated PATHEXT behavior is expecting to find .exe as a suffix. I'll look at the changes to shutil.which in 3.12 to see if I can understand it better by going source diving.

@webknjaz
Copy link
Member

@sirosen sounds like that's solvable by appending .exe? At least, this should demonstrate if the assumption is correct.

@hugovk
Copy link
Member

hugovk commented Jun 16, 2025

Also add the 3.12 classifier:

https://github.com/jazzband/pip-tools/blob/e4ed0c1e028d1ca73673a51722ba153f0c02b0c6/pyproject.toml#L22-LL26

sirosen and others added 3 commits June 16, 2025 17:04
- Add 3.12 to CI
- Update a test which fails on 3.12 due to changes in CPython/pip
  behaviors, with a comment explaining why
- Update the `tox` `piplowest` build to now select `pip==23.2.*`
  on Python 3.12+.

The lowest declared supported `pip` version (`22.2.*`) does not work
on 3.12.

This change builds on previous efforts by other pip-tools
contributors:
- jazzband#2148
- jazzband#2183

Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <578543+webknjaz@users.noreply.github.com>
Co-authored-by: Andy Kluger <1787385+AndydeCleyre@users.noreply.github.com>
On Python 3.12+, `shutil.which()` will not recognize executables which
do not match `PATHEXT` even when given an absolute path to an
executable. As a result, this stub executable used in a test case
fails to be recognized and an unexpected error is emitted.
Also, reorder the classifiers to be in ascending order.

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
@sirosen sirosen force-pushed the support-python312 branch from 0f210b3 to 571cbcd Compare June 16, 2025 22:07
@sirosen
Copy link
Member Author

sirosen commented Jun 16, 2025

@sirosen sounds like that's solvable by appending .exe? At least, this should demonstrate if the assumption is correct.

That was my expectation, and I tried to verify it by pushing a test commit to my fork (separate branch) to run CI with this.
I've pushed the change here as well.

I don't "fully get it" because it seems to me like an absolute path ought to succeed if there's an executable file there? But I'm not very knowledgeable about Windows, so maybe this PATHEXT behavior in shutil.which() is really more correct.

It seems to work, but I lost a bit of time not trusting the fix because the CI output on success was not what I expected to see. We're only getting the successful run of a tox environment reported, not the pytest output. I'm not clear on why that is.


I've also modified the primary commit for 3.12 support with co-author info + added a change co-authored by Hugo for the classifier.

Assuming this passes in CI, I feel happy with the result!

@sirosen sirosen added the enhancement Improvements to functionality label Jun 16, 2025
@sirosen sirosen added this to the 7.4.2 milestone Jun 16, 2025
@sirosen sirosen requested a review from webknjaz June 18, 2025 17:50
@webknjaz
Copy link
Member

We're only getting the successful run of a tox environment reported, not the pytest output. I'm not clear on why that is.

@sirosen wow! This is bizarre! I didn't realize. I don't see an obvious explanation for this behavior.

This may be tox-dev/tox#3193, but I thought it was addressed already. It's not been happening in most of my other projects, but I'm seeing https://github.com/cherrypy/cheroot/actions/runs/15088798220/job/42414767484#step:16:1 using a command with more CLI args, perhaps that's why.

Could you validate this guess?

Copy link
Member

@webknjaz webknjaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sirosen I'm approving this but leaving merging itself to you. There's an opportunity to accept a suggested change + maybe squash some related commits if you'd like to do that before merging.

@webknjaz
Copy link
Member

We're only getting the successful run of a tox environment reported, not the pytest output. I'm not clear on why that is.

@sirosen wow! This is bizarre! I didn't realize. I don't see an obvious explanation for this behavior.

This may be tox-dev/tox#3193, but I thought it was addressed already. It's not been happening in most of my other projects, but I'm seeing cherrypy/cheroot/actions/runs/15088798220/job/42414767484#step:16:1 using a command with more CLI args, perhaps that's why.

Could you validate this guess?

Yep. That was it. We can merge #2190 right after merging this PR.

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
@sirosen
Copy link
Member Author

sirosen commented Jun 19, 2025

Excellent job tracking that down! I was thinking it looked like parallel output but I didn't see where we're setting that. I'll wait for CI and then we should be able to merge this one.

@sirosen sirosen added this pull request to the merge queue Jun 19, 2025
Merged via the queue into jazzband:main with commit 8d5e46e Jun 19, 2025
39 checks passed
@sirosen sirosen deleted the support-python312 branch June 19, 2025 15:11
@sirosen sirosen mentioned this pull request Jun 27, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvements to functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants