This is a followup on #27824: spkg-configure.m4 for python3.

In the next step, we make it possible for users to enable use of system Python packages (site packages).

It is crucial to distinguish two categories of Python packages.

A. Python-implemented applications not running in the same process as sagelib

For some specific packages that provide Python-implemented applications and do not run in the same process as sagelib, there are separate tickets that may use spkg-configure.m4

• System package information, spkg-configure for Jupyter "notebook" package, "rst2ipynb", and dependencies #30124 enables use of system packages for the Python packages notebook (System package information, spkg-configure for Jupyter "notebook" package, "rst2ipynb", and dependencies #30124), rst2ipynb, and their dependencies.

'''B. Packages that run in the same process as sagelib.``

Use of these packages would be enabled by a new configure option --enable-system-site-packages.

This will set up the venv using build/bin/sage-venv --system-site-packages

Approach 1:

• configure --enable-system-site-packages via spkg-configure.m4 for cvxopt and other python packages #29665: configure --enable-system-site-packages via spkg-configure.m4

Approach 2:

• Separate the installation process of a Python package into several phases (makefile targets):

  i) Downloading the spkg tarball (in most cases the tarball will already be an sdist, i.e., with embedded metadata)

  ii) (When patches are needed or the upstream tarball is not a real sdist:) Unpack, patch, make a new sdist

  iii) Copy the sdist to $SAGE_SPKG_WHEELS or a new directory next to it

  iv) Build a wheel and copy to $SAGE_SPKG_WHEELS

  v) Install the wheel

Questions to be resolved:

• #32492 comment:17 -- installed packages are only considered by pip if they can also be found in an index. As we use --no-index, the workaround in sage-pip-install: Try installing without --no-deps first #32492 provides the directory $SAGE_SPKG_WHEELS as the index.

  To be checked if the specific installed version of the package needs to be found in the index. If presence of any version (= spkg version) will cause pip to consider the installed version as a candidate for resolution, then we can just leave the decision whether to use the installed package or the spkg to the resolver and the pip upgrade-strategy.

  If a package needs any special preparation for building a wheel such as setting of environment variables, we need to:

  a) either invoke phase iv) explicitly (so wheels of such packages may be built but remain unused);

  b) or try to move the environment settings to the general build environment;

  c) or patch the Python package's build script instead, treating it like phase ii).

• If we need a specific version (with patches that have not been upstreamed), should this be reflected in dependency information (install-requires.txt)?

• Investigate pip's and other tools behavior when installing an upgraded/downgraded version of a package in the venv when the package is installed in the system site packages

Related/follow-up tickets:

• Meta-ticket: Eliminate use of patches for Python packages #31543 Meta-ticket: Eliminate use of patches for Python packages
• sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}" #29041 at ./bootstrap time, generate requirements.txt, constraints.txt, setup.cfg [install_requires], Pipfile from build/pkgs
• Add src/requirements.txt for installation of sagelib in a venv #30578 Add src/requirements.txt and src/setup.cfg [install_requires] for installation of sagelib in a venv
• Use pipenv to create venv for development #30371 In-place (editable) installs of sagelib in a venv

Tickets/issues for individual packages (moved here from #27330, will be obsolete with this ticket):

• numpy
• scipy
• cypari (depends on pari)

References:

https://packaging.python.org/discussions/install-requires-vs-requirements/

PEP 508 -- Dependency specification for Python Software Packages | Python.org​
https://www.python.org/dev/peps/pep-0508/#pep440

PEP 440 -- Version Identification and Dependency Specification | Python.org
​https://www.python.org/dev/peps/pep-0440/#local-version-segments

https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies

CC: @tobihan @sagetrac-tmonteil @orlitzky @timokau @kiwifb @isuruf @mezzarobba @embray @dimpase @thierry-FreeBSD @dkwo

Component: packages: standard

Issue created by migration from https://trac.sagemath.org/ticket/29023