Find and uninstall broken installed SPKGs and wheels #34203
Description
When system packages are upgraded, shared libraries that a Sage installation links to can disappear, rendering the Sage installation broken. This is a common problem, reported for example in https://groups.google.com/g/sage-devel/c/Iz8ZsmQM3Pg/m/CPl9CbHmBgAJ
To avoid a full rebuild (make distclean && make build), we can try to find the specific broken packages.
The new command make -j list-broken-packages assists the user with this:
$ make -j list-broken-packages
make --no-print-directory auditwheel_or_delocate-no-deps
sage-logger -p 'sage --pip install -r "/Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt"' '/Users/mkoeppe/s/sage/sage-rebasing/logs/pkgs/auditwheel_or_delocate.log'
[auditwheel_or_delocate] Ignoring auditwheel: markers 'sys_platform != "darwin"' don't match your environment
[auditwheel_or_delocate] Requirement already satisfied: delocate in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from -r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.10.2)
[auditwheel_or_delocate] Requirement already satisfied: wheel>=0.32.0 in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.37.1)
[auditwheel_or_delocate] Requirement already satisfied: typing-extensions in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (3.10.0.0)
make[2]: warning: -jN forced in submake: disabling jobserver mode.
# Checking /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/installed/bliss-0.73+debian-1+sage-2016-08-02.p0
...
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libumfpack.dylib'
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib'
Error during installcheck of 'suitesparse': /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib
Uninstall broken packages by typing:
make antic-SAGE_LOCAL-uninstall;
make e_antic-SAGE_LOCAL-uninstall;
make lcalc-SAGE_LOCAL-uninstall;
make ratpoints-SAGE_LOCAL-uninstall;
make normaliz-SAGE_LOCAL-uninstall;
make r-SAGE_LOCAL-uninstall;
make backports_zoneinfo-SAGE_VENV-uninstall;
make fastjsonschema-SAGE_VENV-uninstall;
make pytz_deprecation_shim-SAGE_VENV-uninstall;
make stack_data-SAGE_VENV-uninstall;
make tinycss2-SAGE_VENV-uninstall;
make tzdata-SAGE_VENV-uninstall;
make suitesparse-SAGE_LOCAL-uninstall;
real 2m25.787s
user 4m54.270s
sys 4m0.796s
The core functionality is implemented by the new command sage-spkg-installcheck (implemented in sage_bootstrap):
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose givaro'
Checking shared library file 'lib/libgivaro.9.dylib'
Checking shared library file 'lib/libgivaro.dylib'
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose matplotlib $SAGE_VENV'
Checking wheel file 'var/lib/sage/wheels/matplotlib-3.5.1-cp310-cp310-macosx_12_0_x86_64.whl'
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose e_antic'
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib'
ERROR [libsana|get_dependencies:137]:
/usr/local/opt/flint/lib/libflint-16.dylib not found:
Needed by: /Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib
Error during installcheck of 'e_antic': Could not find all dependencies.
It inspects the installation records in $SAGE_LOCAL/var/lib/sage/installed/ and all shared libraries and platform wheels listed there.
On macOS, instead of using otool directly, we use delocate-listdeps from https://pypi.org/project/delocate/
On Linux, instead of using ldd directly, we use auditwheel.lddtree.lddtree (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/lddtree.py)
(Both packages are also used by cibuildwheel)
See also: #34205 Optionally run auditwheel/delocate on wheels before installing them
Depends on #29097
CC: @dimpase @jhpalmieri @nbruin @EmmanuelCharpentier
Component: build
Author: Matthias Koeppe
Branch/Commit: 3c70a38
Reviewer: John Palmieri
Issue created by migration from https://trac.sagemath.org/ticket/34203