Clean up setup.py in favor of pyproject.toml #4879

justinchuby · 2023-02-09T20:09:48Z

Description

Move much of the metadata from setup.py to pyproject.toml. Setuptools commands are left unchanged.
Replace --weekly_build with the env var ONNX_PREVIEW_BUILD. Version number verified locally and in CI.

Note
Due to pyproject.toml limitation, we need to use a script to rename the package for the weekly build.
On Windows, it is (Get-Content -Path 'pyproject.toml') | ForEach-Object { $_ -replace 'name = "onnx"', 'name = "onnx-weekly"' } | Set-Content -Path 'pyproject.toml'
On linux, it is sed -i 's/name = "onnx"/name = "onnx-weekly"/' 'pyproject.toml'
Tested locally.
Replace all setup.py xxx usage with pip install or python -m build according to setuptools' suggestions.
The MacOS wheel's platform name needs to be tweaked. Previously it was modified by supplying the -p or --plat-name option to setup.py. Now that we use python -m build, I created an env var ONNX_WHEEL_PLATFORM_NAME and supply it via the setup call:
```
setuptools.setup(
    ...
    options={"bdist_wheel": {"plat_name": ONNX_WHEEL_PLATFORM_NAME}}
    if ONNX_WHEEL_PLATFORM_NAME is not None
    else {},
)
```

Motivation and Context

https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

Starting with PEP 621, the Python community selected pyproject.toml as a standard way of specifying project metadata. Setuptools has adopted this standard and will use the information contained in this file as an input in the build process.

https://pypi.org/project/pytest-runner/

It is recommended that you:
Remove 'pytest-runner' from your setup_requires, preferably removing the setup_requires option.
Remove 'pytest' and any other testing requirements from tests_require, preferably removing the tests_requires option.
Select a tool to bootstrap and then run tests such as tox.

fix #4878 and #4539

pyproject.toml

justinchuby · 2023-02-09T21:02:57Z

cannot import name 'TensorProto' from 'onnx' (unknown location)

package find may need update

justinchuby · 2023-02-09T21:28:29Z

@jcwchen build_ext was not called in setup.py. How does it originally work?

jcwchen · 2023-02-09T21:56:32Z

@jcwchen build_ext was not called in setup.py. How does it originally work?

Good question. Although I am not very familiar with setuptools, I guess by default build_ext will be run under some circumstances?

justinchuby · 2023-02-09T22:58:14Z

Looks like only macOS is failing. Could you trigger the release pipelines?

justinchuby · 2023-02-10T00:16:09Z

[100%] Built target onnx_cpp2py_export
1862
copying /home/runner/work/onnx/onnx/.setuptools-cmake-build/onnx_cpp2py_export.cpython-310-x86_64-linux-gnu.so -> /home/runner/work/onnx/onnx/build/lib.linux-x86_64-3.10/onnx
1863
error: could not create '/home/runner/work/onnx/onnx/build/lib.linux-x86_64-3.10/onnx/onnx_cpp2py_export.cpython-310-x86_64-linux-gnu.so': No such file or directory

jcwchen · 2023-03-16T16:54:28Z

Retrigger release CIs

justinchuby · 2023-03-20T22:00:04Z

              If you are seeing this warning it is very likely that a setuptools
              plugin or customization overrides the `build_py` command, without
              taking into consideration how editable installs run build steps
              starting from v64.0.0.
  
              Plugin authors and developers relying on custom build steps are encouraged
              to update their `build_py` implementation considering the information in
              https://setuptools.pypa.io/en/latest/userguide/extension.html
              about editable installs.
  
              For the time being `setuptools` will silence this error and ignore
              the faulty command, but this behaviour will change in future versions.

Signed-off-by: Justin Chu <justinchu@microsoft.com>

github-advanced-security · 2023-04-05T19:29:45Z

You have successfully added a new lintrunner configuration lintrunner. As part of the setup process, we have scanned this repository and found no existing alerts. In the future, you will see all code scanning alerts on the repository Security tab.

pyproject.toml

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

Signed-off-by: Justin Chu <justinchu@microsoft.com>

requirements-dev.txt

Signed-off-by: Justin Chu <justinchu@microsoft.com>

pyproject.toml

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby · 2023-09-05T15:03:23Z

Bump @jcwchen

jcwchen

One nit comment. Others look good to me.

setup.py

jcwchen · 2023-09-05T15:34:28Z

To confirm, will this PR resolve #4539? Or it will need further fix?

justinchuby · 2023-09-05T15:38:33Z

To confirm, will this PR resolve #4539? Or it will need further fix?

It will fix it

justinchuby · 2023-09-05T15:54:25Z

https://beta.ruff.rs/docs/rules/logging-f-string/

### Description  - Move much of the metadata from setup.py to pyproject.toml. Setuptools commands are left unchanged. - Replace `--weekly_build` with the env var `ONNX_PREVIEW_BUILD`. Version number verified locally and in CI. > **Note** > Due to `pyproject.toml` limitation, we need to use a script to rename the package for the weekly build. > On Windows, it is `(Get-Content -Path 'pyproject.toml') | ForEach-Object { $_ -replace 'name = "onnx"', 'name = "onnx-weekly"' } | Set-Content -Path 'pyproject.toml'` > On linux, it is `sed -i 's/name = "onnx"/name = "onnx-weekly"/' 'pyproject.toml'` > Tested locally. - Replace all `setup.py xxx` usage with `pip install` or `python -m build` according to setuptools' suggestions. - The MacOS wheel's platform name needs to be tweaked. Previously it was modified by supplying the `-p` or `--plat-name` option to `setup.py`. Now that we use `python -m build`, I created an env var `ONNX_WHEEL_PLATFORM_NAME` and supply it via the `setup` call: ``` setuptools.setup( ... options={"bdist_wheel": {"plat_name": ONNX_WHEEL_PLATFORM_NAME}} if ONNX_WHEEL_PLATFORM_NAME is not None else {}, ) ``` ### Motivation and Context   https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html > Starting with [PEP 621](https://peps.python.org/pep-0621/), the Python community selected pyproject.toml as a standard way of specifying project metadata. Setuptools has adopted this standard and will use the information contained in this file as an input in the build process. https://pypi.org/project/pytest-runner/ > It is recommended that you: > Remove 'pytest-runner' from your setup_requires, preferably removing the setup_requires option. > Remove 'pytest' and any other testing requirements from tests_require, preferably removing the tests_requires option. > Select a tool to bootstrap and then run tests such as tox. fix #4878 and #4539 --------- Signed-off-by: Justin Chu <justinchu@microsoft.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby · 2023-09-05T17:20:41Z

I misspoke - editable build support will require #5558 @jcwchen

### Description  - Move much of the metadata from setup.py to pyproject.toml. Setuptools commands are left unchanged. - Replace `--weekly_build` with the env var `ONNX_PREVIEW_BUILD`. Version number verified locally and in CI. > **Note** > Due to `pyproject.toml` limitation, we need to use a script to rename the package for the weekly build. > On Windows, it is `(Get-Content -Path 'pyproject.toml') | ForEach-Object { $_ -replace 'name = "onnx"', 'name = "onnx-weekly"' } | Set-Content -Path 'pyproject.toml'` > On linux, it is `sed -i 's/name = "onnx"/name = "onnx-weekly"/' 'pyproject.toml'` > Tested locally. - Replace all `setup.py xxx` usage with `pip install` or `python -m build` according to setuptools' suggestions. - The MacOS wheel's platform name needs to be tweaked. Previously it was modified by supplying the `-p` or `--plat-name` option to `setup.py`. Now that we use `python -m build`, I created an env var `ONNX_WHEEL_PLATFORM_NAME` and supply it via the `setup` call: ``` setuptools.setup( ... options={"bdist_wheel": {"plat_name": ONNX_WHEEL_PLATFORM_NAME}} if ONNX_WHEEL_PLATFORM_NAME is not None else {}, ) ``` ### Motivation and Context   https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html > Starting with [PEP 621](https://peps.python.org/pep-0621/), the Python community selected pyproject.toml as a standard way of specifying project metadata. Setuptools has adopted this standard and will use the information contained in this file as an input in the build process. https://pypi.org/project/pytest-runner/ > It is recommended that you: > Remove 'pytest-runner' from your setup_requires, preferably removing the setup_requires option. > Remove 'pytest' and any other testing requirements from tests_require, preferably removing the tests_requires option. > Select a tool to bootstrap and then run tests such as tox. fix onnx#4878 and onnx#4539 --------- Signed-off-by: Justin Chu <justinchu@microsoft.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Signed-off-by: Corwin Joy <corwinjoy@gmail.com>

justinchuby force-pushed the justinchu/cleanup-setup branch from e7db237 to aae0d6d Compare February 9, 2023 20:55

justinchuby marked this pull request as ready for review February 9, 2023 20:58

justinchuby requested a review from a team as a code owner February 9, 2023 20:58

justinchuby commented Feb 9, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

justinchuby marked this pull request as draft February 9, 2023 21:03

justinchuby force-pushed the justinchu/cleanup-setup branch from 0b615b7 to f7197d0 Compare February 9, 2023 21:54

justinchuby marked this pull request as ready for review February 9, 2023 23:13

justinchuby marked this pull request as draft March 16, 2023 14:40

justinchuby marked this pull request as ready for review March 16, 2023 14:43

jcwchen added the run release CIs Use this label to trigger release tests in CI label Mar 16, 2023

jcwchen closed this Mar 16, 2023

jcwchen reopened this Mar 16, 2023

justinchuby marked this pull request as draft March 20, 2023 22:03

justinchuby closed this Apr 5, 2023

justinchuby force-pushed the justinchu/cleanup-setup branch from 7157829 to aaf13fd Compare April 5, 2023 18:46

justinchuby reopened this Apr 5, 2023

justinchuby force-pushed the justinchu/cleanup-setup branch from 516312d to b706a06 Compare April 5, 2023 18:47

Clean up setup

a3dd5eb

Signed-off-by: Justin Chu <justinchu@microsoft.com>

justinchuby force-pushed the justinchu/cleanup-setup branch from b706a06 to a3dd5eb Compare April 5, 2023 18:49

justinchuby commented Apr 5, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

justinchuby commented Apr 5, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

Apply suggestions from code review

846d91a

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

justinchuby added 2 commits August 31, 2023 20:48

Rename package

8a62183

Signed-off-by: Justin Chu <justinchu@microsoft.com>

Authors

bfe4a3a

Signed-off-by: Justin Chu <justinchu@microsoft.com>

justinchuby added this to the 1.15 milestone Aug 31, 2023

justinchuby commented Aug 31, 2023

View reviewed changes

requirements-dev.txt Show resolved Hide resolved

Use new_name

5f1434f

Signed-off-by: Justin Chu <justinchu@microsoft.com>

justinchuby mentioned this pull request Aug 31, 2023

Support Python editable builds #5558

Merged

Fix mac platform target

f707986

Signed-off-by: Justin Chu <justinchu@microsoft.com>

justinchuby commented Aug 31, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

Update pyproject.toml

3410002

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>

jcwchen approved these changes Sep 5, 2023

View reviewed changes

setup.py Show resolved Hide resolved

setup.py Show resolved Hide resolved

jcwchen added topic: build Issues related to ONNX builds and packages module: CI pipelines Issues related to the CI pipeline labels Sep 5, 2023

justinchuby enabled auto-merge September 5, 2023 15:38

jcwchen linked an issue Sep 5, 2023 that may be closed by this pull request

Build error with the latest setuptool #4539

Closed

justinchuby added this pull request to the merge queue Sep 5, 2023

Merged via the queue into onnx:main with commit 8e2a216 Sep 5, 2023

justinchuby deleted the justinchu/cleanup-setup branch September 5, 2023 16:42

Merged via the queue into onnx:main with commit bd60a39 Sep 5, 2023

Clean up setup.py in favor of pyproject.toml #4879

Clean up setup.py in favor of pyproject.toml #4879

Uh oh!

Conversation

justinchuby commented Feb 9, 2023 • edited by jcwchen Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

Uh oh!

justinchuby commented Feb 9, 2023

Uh oh!

justinchuby commented Feb 9, 2023

Uh oh!

jcwchen commented Feb 9, 2023

Uh oh!

justinchuby commented Feb 9, 2023

Uh oh!

justinchuby commented Feb 10, 2023

Uh oh!

jcwchen commented Mar 16, 2023

Uh oh!

justinchuby commented Mar 20, 2023

Uh oh!

github-advanced-security bot commented Apr 5, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

justinchuby commented Sep 5, 2023

Uh oh!

jcwchen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jcwchen commented Sep 5, 2023

Uh oh!

justinchuby commented Sep 5, 2023

Uh oh!

justinchuby commented Sep 5, 2023

Uh oh!

justinchuby commented Sep 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

justinchuby commented Feb 9, 2023 •

edited by jcwchen

Loading

justinchuby commented Sep 5, 2023 •

edited

Loading