(from #29146, #30861)

There are several places where the spkg (build/pkgs/*/) and system package information (build/pkgs/*/distros/*.txt) is parsed:

• bootstrap
• src/doc/bootstrap
• build/bin/write_dockerfile.sh
• build/bin/sage-get-system-packages
• tox.ini (for local-homebrew, local-conda)
• .github/workflows/ci-wsl.yml (using powershell)
• src/bin/sage-download-upstream
• m4/sage_spkg_*.m4

The purpose of this ticket is to prepare refactoring of this code by extending the sage_bootstrap package.

• We update sage_bootstrap.package so that it understands non-normal packages (i.e., script and pip packages, which do not have checksums.ini)
• We change sage -package list so it supports the following invocations:
  • sage -package list :standard: :optional: (several "classes" (types), not just one)
  • sage -package list --has-file=spkg-configure.m4 :standard: (needed for bootstrap, src/doc/bootstrap)
  • sage -package list --has-file=spkg-configure.m4 --has-file=distros/debian.txt :standard: :optional: (needed for build/bin/write-dockerfile.sh, tox.ini)
  • sage -package list --has-file=SPKG.rst :all: (needed for src/doc/bootstrap)
• We fix and extend sage -package download (follow up from sage --package download is broken #30648) so it supports
  • sage -package download :all: (again - was broken)
  • sage -package download --allow-upstream :all: (allows retrieving from upstream_url)
    and use it for make download (fix make download #29896), removing src/bin/sage-download-upstream

To run the testsuite of sage_bootstrap, use

  (cd build && tox)

Follow-up:

• The actual refactoring of the above scripts (src/doc/bootstrap: Simplify by using new options of "sage -package list" #30947, tox.ini, build/bin/write-dockerfile.sh: Simplify by using new options of "sage -package list" #30951, bootstrap: Simplify by using new options of "sage -package list" #30968, ...)
• Wishlist item: sage -package list --not-source=pip :optional: :experimental: (would simplify bootstrap)

Depends on #30648

CC: @seblabbe @tobiasdiez @slel @jhpalmieri @vbraun

Component: build: configure

Author: Matthias Koeppe

Branch: f33c363

Reviewer: Sébastien Labbé

Issue created by migration from https://trac.sagemath.org/ticket/30865