Our current version of pip already tries to install packages via bdist_wheel.

In this ticket, we break the build process into these steps by modifying sdh_pip_install:

1. Build the wheel explicitly using pip wheel --no-binary :all: --no-build-isolation -w "$SAGE_SPKG_WHEELS" (or setup.py bdist_wheel), which stores the wheel in SAGE_SPKG_WHEELS=$SAGE_LOCAL/var/lib/sage/wheels.

2. Then install the wheel.

The wheels in $SAGE_SPKG_WHEELS persist after the completed build. We manage them using the DESTDIR staging mechanism -- there will be exactly 1 wheel for each installed package (and removing a package removes the wheel). Example (after rebuilding a few packages on this branch):

$ ls -l local/var/lib/sage/wheels/
...
-rw-r--r--  1 mkoeppe  staff    545293 Sep  8 12:44 Pillow-7.2.0-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff   4573510 Sep  8 12:06 numpy-1.19.1-cp37-cp37m-macosx_10_9_x86_64.whl
-rw-r--r--  1 mkoeppe  staff      2237 Sep  8 12:44 sage_conf-9.2b12-py3-none-any.whl
-rw-r--r--  1 mkoeppe  staff  17981103 Sep  8 12:59 scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl

Users can create virtual environments and install the wheels built by the Sage distribution into them, using standard tools such as pip install --find-links.

• Example:

  ```
  $ python3 -m venv some-venv
  $ cd some-venv/
  $ bin/pip3 install -v --no-index --find-links=../local/var/lib/sage/wheels/ Pillow    
  

• In follow-up ticket Create PEP 503 simple repository for wheels built during installation #30527, we create a PEP 503 simple repository for the wheels built during installation, which enables more convenient installation options for users.

In this ticket, we keep using the DESTDIR staging mechanism also for the installation of the package from the wheel.

• In a follow-up ticket, we can change this -- to remove the problems described in Remove DESTDIR staging for Python packages to eliminate race conditions during Python package installations #29585 -- by just holding the pip lock (if it is at all still necessary after the major pip upgrades in the 9.2 series) during the wheel installation (step 2).

References:

• https://pip.pypa.io/en/stable/user_guide/#installing-from-local-packages

• https://pip.pypa.io/en/stable/reference/pip_cache/

CC: @slel @tobiasdiez @embray @jhpalmieri @tscrim

Component: build

Author: Matthias Koeppe

Branch: f2e7075

Reviewer: Tobias Diez, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/29500