Document building from the monorepo using `PIP_CONSTRAINT=$SAGE_ROOT/constraints_pkgs.txt` #37434

mkoeppe · 2024-02-23T06:55:20Z

This is aimed at the intermediate level, for users who wish to build a set of distribution packages as supplied in pkgs/ but without using the wheelhouse infrastructure of Sage-the-distribution.

Documentation included in the files.

It can also be tested using new tox environments defined in pkgs/sagemath-standard/tox.ini, and is tested on GH Actions: https://github.com/mkoeppe/sage/actions/workflows/ci-linux-incremental.yml?query=branch%3Aconstraints_pkgs

this also provides the missing automatic test for the sagemath-standard distribution built out of pkgs/sagemath-standard/

Marked as critical because it is hoped to ease the adoption of modularized distribution packages by downstream packagers.

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

Depends on src/pyproject.toml: Add missing build dep jinja2 #38515 (merged here)
Depends on cython 3.0.11 compatibility #38500 (merged here)

src/pyproject.toml.m4

github-actions · 2024-04-01T10:14:29Z

Documentation preview for this PR (built with commit 3ab2a18; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

kiwifb · 2024-04-15T04:47:53Z

Yes, I understand how this would have failed if the sdist had no .py files like it is happening for 10.3.beta4 right now. For that reason, CI will probably fail if we commit this PR now. I cannot see any obvious problem. I would consider a clean CI run to be enough to decide this can be included.

pkgs/sagemath-standard/tox.ini: Add environment sagepython-constraints_pkgs[-norequirements]

…ts_pkgs-norequirements

…equirements

For C-return-type functions that raise exceptions (and their heirs).

kwankyu · 2024-08-16T01:57:19Z

I get

$ export PIP_CONSTRAINT="$(pwd)/constraints_pkgs.txt"
$ ./sage -sh -c 'python3 -m build pkgs/sagemath-standard'
...
  File "/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/build-via-sdist-pj7cyyv6/sagemath_standard-10.5b2/sage_setup/autogen/interpreters/utils.py", line 19, in <module>
    from jinja2 import Environment
ModuleNotFoundError: No module named 'jinja2'
************************************************************************
Error building the Sage library
************************************************************************

mkoeppe · 2024-08-16T02:33:19Z

Thanks for testing. ~~I think I have repaired this breakage already somewhere else~~ Fix in #38515

… into constraints_pkgs

…ts_pkgs-norequirements' runs

…constraints_pkgs-norequirements' runs

…_pkgs-norequirements' runs

kwankyu · 2024-08-16T09:10:01Z

The wheel was built successfully. But pip-installing it in a venv, I get

...
...
Using cached pure_eval-0.2.3-py3-none-any.whl (11 kB)
Using cached wcwidth-0.2.13-py2.py3-none-any.whl (34 kB)
Building wheels for collected packages: fpylll, lrcalc
  Building wheel for fpylll (pyproject.toml) ... -done
  Created wheel for fpylll: filename=fpylll-0.6.1-cp39-cp39-macosx_12_0_x86_64.whl size=879911 sha256=62fa81e4cfb35635b0804f3691ff603ebadae4be7f11c3a127fb8a0b9fa3a0d1
  Stored in directory: /Users/kwankyu/Library/Caches/pip/wheels/9d/f5/89/488429ee43f2e24a12435aa9fb258eeb5bd1f27cfd5e35710b
  Building wheel for lrcalc (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for lrcalc (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-build-env-phum_0fo/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py:268: UserWarning: Unknown distribution option: 'long_description_type'
        warnings.warn(msg)
      running bdist_wheel
      running build
      running build_ext
      building 'lrcalc' extension
      creating build
      creating build/temp.macosx-12-x86_64-cpython-39
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/kwankyu/Venv/python39/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c lrcalc.c -o build/temp.macosx-12-x86_64-cpython-39/lrcalc.o
      lrcalc.c:1239:10: fatal error: 'lrcalc/ivector.h' file not found
      #include "lrcalc/ivector.h"
               ^~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lrcalc
Successfully built fpylll
Failed to build lrcalc
ERROR: Could not build wheels for lrcalc, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip

kwankyu · 2024-08-16T09:18:31Z

~~This seems like a problem of my system though.~~ I don't know.

mkoeppe · 2024-08-16T17:08:17Z

Did you run it within sage -sh? That's needed so that the lrcalc Python bindings can find the lrcalc library -- unless you have a systemwide installation of it.

… for 'constraints_pkgs-norequirements' runs

mkoeppe · 2024-08-16T17:18:38Z

I've also noted in #31251 that it would be desirable for lrcalc upstream to publish wheels to PyPI.

kwankyu · 2024-08-17T00:30:31Z

Did you run it within sage -sh? That's needed so that the lrcalc Python bindings can find the lrcalc library -- unless you have a systemwide installation of it.

No. I ran it in a new venv. What is the point of installing the sagemath-standard wheel within sage -sh?

In the venv, I get

(python39) Hera:~$ pip install lrcalc
Collecting lrcalc
  Using cached lrcalc-2.1.tar.gz (16 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: lrcalc
  Building wheel for lrcalc (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for lrcalc (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      /private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-build-env-5pnlind4/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py:268: UserWarning: Unknown distribution option: 'long_description_type'
        warnings.warn(msg)
      running bdist_wheel
      running build
      running build_ext
      building 'lrcalc' extension
      creating build
      creating build/temp.macosx-12-x86_64-cpython-39
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/kwankyu/Venv/python39/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c lrcalc.c -o build/temp.macosx-12-x86_64-cpython-39/lrcalc.o
      lrcalc.c:1239:10: fatal error: 'lrcalc/ivector.h' file not found
      #include "lrcalc/ivector.h"
               ^~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lrcalc
Failed to build lrcalc
ERROR: Could not build wheels for lrcalc, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip

kwankyu · 2024-08-17T01:10:36Z

I've also noted in #31251 that it would be desirable for lrcalc upstream to publish wheels to PyPI.

So all the fuss I went through is unavoidable unless lrcalc is published to PyPI?

mkoeppe · 2024-08-17T01:18:57Z

Did you run it within sage -sh? That's needed so that the lrcalc Python bindings can find the lrcalc library -- unless you have a systemwide installation of it.

No. I ran it in a new venv. What is the point of installing the sagemath-standard wheel within sage -sh?

Within sage -sh, environment variables are set that are needed to discover the libraries installed in SAGE_LOCAL.

You can activate your venv within the sage -sh.

kwankyu · 2024-08-17T01:24:54Z

I see. My venvs are outside of sage -sh. There was no problem except lrcalc.

kwankyu · 2024-08-17T01:27:49Z

The step pip install lrcalc was not really necessary.

This is the first time that I successfully installed sagemath-standard distribution package. Personally, it is the most impressive product of the modularization efforts :-)

kwankyu · 2024-08-17T13:05:42Z

Perhaps not an issue directly related with this PR, but I wonder if lrcalc (and others if any) is a prerequisite of sagemath-standard and if that should be documented somewhere.

mkoeppe · 2024-08-17T17:20:57Z

Basically all standard packages are dependencies of sagemath-standard, so https://doc.sagemath.org/html/en/reference/spkg/index.html#standard-packages provides this documentation.

The direct non-Python dependencies of sagemath-standard are also declared in https://github.com/sagemath/sage/blob/develop/src/pyproject.toml#L25, but this is at the moment only decoration because there is no tooling that reads this information.

Note that the user-facing documentation of installation with Python tools is in https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi - it provisions the non-Python dependencies via sage_conf.

kwankyu · 2024-08-18T00:24:10Z

Note that the user-facing documentation of installation with Python tools is in https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi - it provisions the non-Python dependencies via sage_conf.

So Python dependencies are automatically resolved and installed via pip, and non-Python dependencies should be provisioned by the system or by sage_conf. For my case, my system only lacked lrcalc, rather fortunately.

https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi perfectly explains the installation procedure of sagemath-standard. I think that there should be a link to it or a copy in the Project description of sagemath-standard at https://pypi.org/project/sagemath-standard.

…emath/sage/blob/develop/README.md\#alternative-installation-using-pypi

mkoeppe · 2024-08-18T00:34:03Z

Good idea, done in 02f6e88

pkgs/sagemath-standard/constraints_pkgs.txt

…experimental'

…ecific to sagemath-standard

…ecific to sagemath-standard (fixup)

kwankyu · 2024-08-18T02:29:09Z

Perhaps another test run of the changed workflows?

Otherwise lgtm.

mkoeppe · 2024-08-18T03:06:52Z

Thanks. I think I'll iterate on the workflows in a follow up, when changes such as #38518 have been merged

kwankyu · 2024-08-18T04:08:01Z

OK, as the changes on the workflows are just additions and can be tested on the battleground.

mkoeppe · 2024-08-18T04:38:45Z

Thanks for the review!

sagemathgh-37434: Document building from the monorepo using `PIP_CONSTRAINT=$SAGE_ROOT/constraints_pkgs.txt`   This is aimed at the intermediate level, for users who wish to build a set of distribution packages as supplied in `pkgs/` but without using the wheelhouse infrastructure of Sage-the-distribution. Documentation included in the files. It can also be tested using new tox environments defined in `pkgs/sagemath-standard/tox.ini`, and is tested on GH Actions: https://github.com/mkoeppe/sage/actions/workflows/ci-linux- incremental.yml?query=branch%3Aconstraints_pkgs - this also provides the missing automatic test for the **sagemath- standard** distribution built out of `pkgs/sagemath-standard/` Marked as critical because it is hoped to ease the adoption of modularized distribution packages by downstream packagers.    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   - Depends on sagemath#38515 (merged here) - Depends on sagemath#38500 (merged here) URL: sagemath#37434 Reported by: Matthias Köppe Reviewer(s): François Bissey, Kwankyu Lee, Matthias Köppe

sagemathgh-38515: `src/pyproject.toml`: Add missing build dep `jinja2`       - Broken in sagemath#37287 - Reported in sagemath#37434 (comment) - Reported in sagemath#36524 (comment) ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38515 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

sagemathgh-38518: CI Linux: Consolidate jobs       Thanks to the more powerful runners in use since early 2024, we can do away with the splits between - `standard-pre` and `standard` - `minimal-pre` and `minimal` - `optional-0-o` and `optional-p-z` - `experimental-0-o` and `experimental-p-z` This makes the CI runs a bit easier to navigate for those who look at them. - Test run: https://github.com/mkoeppe/sage/actions/runs/10438362191 This should also fix "No space left on device" aborts as seen for example in https://github.com/mkoeppe/sage/actions/runs/10426436105 for standard-pre (where we did not use to maximize the available disk space). ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    - Depends on sagemath#37434 (merged here) URL: sagemath#38518 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

sagemathgh-38515: `src/pyproject.toml`: Add missing build dep `jinja2`       - Broken in sagemath#37287 - Reported in sagemath#37434 (comment) - Reported in sagemath#36524 (comment) ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38515 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

sagemathgh-38518: CI Linux: Consolidate jobs       Thanks to the more powerful runners in use since early 2024, we can do away with the splits between - `standard-pre` and `standard` - `minimal-pre` and `minimal` - `optional-0-o` and `optional-p-z` - `experimental-0-o` and `experimental-p-z` This makes the CI runs a bit easier to navigate for those who look at them. - Test run: https://github.com/mkoeppe/sage/actions/runs/10438362191 This should also fix "No space left on device" aborts as seen for example in https://github.com/mkoeppe/sage/actions/runs/10426436105 for standard-pre (where we did not use to maximize the available disk space). ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    - Depends on sagemath#37434 (merged here) URL: sagemath#38518 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

mkoeppe commented Feb 23, 2024

View reviewed changes

src/pyproject.toml.m4 Outdated Show resolved Hide resolved

github-actions bot added the v: moderate label May 31, 2024

vbraun force-pushed the develop branch from eba5e19 to e5f42fa Compare June 3, 2024 22:15

github-actions bot removed the s: needs work label Jul 25, 2024

kiwifb and others added 7 commits August 12, 2024 12:57

Add noexcept in matroids and set for cython 3.0.11 compatibility

c4d3b66

constraints_pkgs.txt, pkgs/sagemath-standard/constraints_pkgs.txt: New

264b913

pkgs/sagemath-standard/tox.ini: Add environment sagepython-constraints_pkgs[-norequirements]

.github/workflows/ci-linux.yml: Test sagelib-tox-sagepython-constrain…

9d5322c

…ts_pkgs-norequirements

.github/workflows/ci-linux-incremental.yml: Test constraints_pkgs-nor…

7598576

…equirements

.github/workflows/ci-linux.yml: Fixup

f8f7b0f

make SPKG-tox-...: Do set the sage environment

114b53c

Add some excepts (#1)

31d6a6b

For C-return-type functions that raise exceptions (and their heirs).

src/pyproject.toml: Add missing build dep jinja2

c52ad79

github-actions bot removed the s: needs work label Aug 16, 2024

Matthias Koeppe added 7 commits August 15, 2024 19:59

Merge branch 'src_pyproject_toml_jinja2' into constraints_pkgs

4eff0a0

pkgs/sagemath-standard/constraints_pkgs.txt: More verbosity in example

b3fd6af

constraints_pkgs.txt: More verbosity in example

61ace88

Merge commit 'refs/pull/38500/head' of https://github.com/sagemath/sage…

b0a6880

… into constraints_pkgs

.github/workflows/ci-linux-incremental.yml: Update, reduce 'constrain…

df5e38a

…ts_pkgs-norequirements' runs

.github/workflows/ci-linux[-incremental].yml: Use 'targets-pre' for '…

b7ff066

…constraints_pkgs-norequirements' runs

.github/workflows/ci-linux.yml: Build all-sage-local for 'constraints…

bfb531e

…_pkgs-norequirements' runs

.github/workflows/ci-linux[-incremental].yml: Ensure tox is installed…

bf00218

… for 'constraints_pkgs-norequirements' runs

github-actions bot added the s: needs review label Aug 17, 2024

pkgs/sagemath-standard/README.rst: Add link to https://github.com/sag…

02f6e88

…emath/sage/blob/develop/README.md\#alternative-installation-using-pypi

kwankyu reviewed Aug 18, 2024

View reviewed changes

pkgs/sagemath-standard/constraints_pkgs.txt Show resolved Hide resolved

Matthias Koeppe added 4 commits August 17, 2024 17:45

pkgs/sagemath-standard/README.rst: Update copyright year; no longer '…

524dc88

…experimental'

pkgs/sagemath-standard/constraints_pkgs.txt: Make explanation more sp…

368e9f6

…ecific to sagemath-standard

pkgs/sagemath-standard/constraints_pkgs.txt: Make explanation more sp…

4204f2d

…ecific to sagemath-standard (fixup)

pkgs/sagemath-standard/constraints_pkgs.txt: Change to just 'Example:'

3ab2a18

kwankyu approved these changes Aug 18, 2024

View reviewed changes

kwankyu added s: positive review and removed s: needs review labels Aug 18, 2024

kwankyu mentioned this pull request Aug 18, 2024

CI Linux: Consolidate jobs #38518

Merged

5 tasks

vbraun merged commit 5ee375c into sagemath:develop Sep 3, 2024
17 of 20 checks passed

github-actions bot removed the s: positive review label Sep 4, 2024

Uh oh!

Document building from the monorepo using PIP_CONSTRAINT=$SAGE_ROOT/constraints_pkgs.txt #37434

Document building from the monorepo using PIP_CONSTRAINT=$SAGE_ROOT/constraints_pkgs.txt #37434

Uh oh!

Conversation

mkoeppe commented Feb 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Checklist

⌛ Dependencies

Uh oh!

Uh oh!

github-actions bot commented Apr 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kiwifb commented Apr 15, 2024

Uh oh!

kwankyu commented Aug 16, 2024

Uh oh!

mkoeppe commented Aug 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwankyu commented Aug 16, 2024

Uh oh!

kwankyu commented Aug 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkoeppe commented Aug 16, 2024

Uh oh!

mkoeppe commented Aug 16, 2024

Uh oh!

kwankyu commented Aug 17, 2024

Uh oh!

kwankyu commented Aug 17, 2024

Uh oh!

mkoeppe commented Aug 17, 2024

Uh oh!

kwankyu commented Aug 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwankyu commented Aug 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwankyu commented Aug 17, 2024

Uh oh!

mkoeppe commented Aug 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwankyu commented Aug 18, 2024

Uh oh!

mkoeppe commented Aug 18, 2024

Uh oh!

Uh oh!

kwankyu commented Aug 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkoeppe commented Aug 18, 2024

Uh oh!

kwankyu commented Aug 18, 2024

Uh oh!

mkoeppe commented Aug 18, 2024

Uh oh!

Uh oh!

Uh oh!

Document building from the monorepo using `PIP_CONSTRAINT=$SAGE_ROOT/constraints_pkgs.txt` #37434

Document building from the monorepo using `PIP_CONSTRAINT=$SAGE_ROOT/constraints_pkgs.txt` #37434

mkoeppe commented Feb 23, 2024 •

edited

Loading

github-actions bot commented Apr 1, 2024 •

edited

Loading

mkoeppe commented Aug 16, 2024 •

edited

Loading

kwankyu commented Aug 16, 2024 •

edited

Loading

kwankyu commented Aug 17, 2024 •

edited

Loading

kwankyu commented Aug 17, 2024 •

edited

Loading

mkoeppe commented Aug 17, 2024 •

edited

Loading

kwankyu commented Aug 18, 2024 •

edited

Loading