Remove DESTDIR staging for Python packages to eliminate race conditions during Python package installations #29585
Description
Follow-up from #26018:
Declaring some Python packages as PYTHON_TOOLCHAIN helped somewhat, but there are more race conditions with the same failure mechanism: Some Python package byte-compiles modules from a half-installed package, and then finishing the installation of that fails.
Of course, the high parallelization in my automatic runs on GitHub provoke these kinds of errors. But this should be fixed nevertheless.
See https://github.com/mkoeppe/sage/runs/618645992
[backports_shutil_get_terminal_size-1.0.0.p1] cp: cannot create regular file '/sage/local/./lib/python2.7/site-packages/backports/__init__.py': File exists
[backports_shutil_get_terminal_size-1.0.0.p1] ************************************************************************
[backports_shutil_get_terminal_size-1.0.0.p1] Error copying files for backports_shutil_get_terminal_size-1.0.0.p1.
[backports_shutil_get_terminal_size-1.0.0.p1] ************************************************************************
[backports_ssl_match_hostname-3.5.0.1.p0] error installing, exit status 1. End of log file:
[backports_shutil_get_terminal_size-1.0.0.p1] Full log file: /sage/logs/pkgs/backports_shutil_get_terminal_size-1.0.0.p1.log
[backports_ssl_match_hostname-3.5.0.1.p0] Copying package files from temporary location /sage/local/var/tmp/sage/build/backports_ssl_match_hostname-3.5.0.1.p0/inst to /sage/local
[backports_ssl_match_hostname-3.5.0.1.p0] cp: cannot create regular file '/sage/local/./lib/python2.7/site-packages/backports/__init__.pyc': File exists
[backports_ssl_match_hostname-3.5.0.1.p0] ************************************************************************
[backports_ssl_match_hostname-3.5.0.1.p0] Error copying files for backports_ssl_match_hostname-3.5.0.1.p0.
[backports_ssl_match_hostname-3.5.0.1.p0] ************************************************************************
[backports_ssl_match_hostname-3.5.0.1.p0] Full log file: /sage/logs/pkgs/backports_ssl_match_hostname-3.5.0.1.p0.log
Another occurrence on Cygwin: https://github.com/mkoeppe/sage/runs/638920131?check_suite_focus=true
Another occurrence: #30831
A new one thanks to recent pip's opportunistic use of tornado:
[tornado-6.0.4] real 0m11.342s
[tornado-6.0.4] user 0m2.041s
[tornado-6.0.4] sys 0m0.380s
[tornado-6.0.4] Copying package files from temporary location /sage/local/var/tmp/sage/build/tornado-6.0.4/inst to /sage/local
[tornado-6.0.4] cp: cannot create directory '/sage/local/./lib/python3.9/site-packages/tornado/__pycache__': File exists
[tornado-6.0.4] ************************************************************************
https://github.com/sagemath/sage/runs/2879682001
In this ticket, we remove the DESTDIR staging for Python packages. It has been redundant since we moved to installing packages via wheels.
After this change, the only file that is tracked in $SAGE_LOCAL/var/lib/sage/installed/pynormaliz-2.14 is the wheel file:
"files": [
"var/lib/sage/wheels/PyNormaliz-2.14-cp39-cp39-macosx_11_0_x86_64.whl"
]
}
Uninstalling Python packages is now done via pip, by means of a new removal script called spkg-piprm installed in $SAGE_LOCAL/var/lib/sage/scripts/SPKG/.
CC: @jhpalmieri @dimpase @embray @kliem
Component: build
Keywords: sd111
Author: Matthias Koeppe
Branch: c3eeb69
Reviewer: John Palmieri
Issue created by migration from https://trac.sagemath.org/ticket/29585