Since Sage 9.4, it is possible to use an installation tree for Python packages separate from SAGE_LOCAL:
https://wiki.sagemath.org/ReleaseTours/sage-9.4#For_developers:_..2Fconfigure_--prefix.3DSAGE_LOCAL_--with-sage-venv.3DSAGE_VENV

In this ticket we make a new setting --with-sage-venv=auto the default, which means:

• if --prefix has not been used (so SAGE_LOCAL=local), then set SAGE_VENV to local/var/lib/sage/venv-PYTHONVERSION
  Users can also pass --with-sage-venv or --with-sage-venv=yes, which means to unconditionally set SAGE_VENV to SAGE_LOCAL/var/lib/sage/venv-PYTHONVERSION.

(We keep the layout as is in incremental builds.)

By keying the default name to the Python version, we ensure that if the system python version changes (either because of system updates or explicit reconfiguration), we create a fresh venv and automatically rebuild all Python packages, eliminating problems such as the one reported in https://groups.google.com/g/sage-devel/c/hZPHxqn_Cyk/m/fOAZCUWHAQAJ

Moreover, at config.status time, we create a symbolic link SAGE_ROOT/venv -> SAGE_VENV (overwriting an existing symbolic link)

The link is for easy access by users and is not used otherwise. Instead of local/lib/python3/site-packages/ or local/bin/sage, use venv/local/lib/python3/site-packages/ etc.

For symmetry, we also create a symbolic link SAGE_ROOT/prefix -> SAGE_LOCAL. (This generalizes what is already done by tox local.)

Users will still be able to restore the previous behavior by using --with-sage-venv=no, or use a specific path as in --with-sage-venv=/path/to/venv.

CC: @dimpase @jhpalmieri

Component: build: configure

Author: Matthias Koeppe

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/32442