The Sage distribution contains a number of "database packages", many of which do not seem to have a "real upstream" (or installation procedures other than our scripts in build/pkgs).

We transform them into pip-installable packages and publish them to PyPI so that they can be installed using standard Python tools (and become Python dependencies of sagelib) and then discovered by libraries using standard Python facilities.

Role model: https://pypi.org/project/database-knotinfo/

For those that do not have a real upstream, we create separate git repositories in github.com/sagemath/ to serve as new upstream.

List of packages/tickets:

Architecture independent:

• database_stein_watkins (New upstream for database_stein_watkins #33881)
• combinatorial_designs (single text file MOLS_table.txt), 8 kB, https://repology.org/project/sagemath-combinatorial-designs/versions
• graphs (graphs.db, brouwer_srg_database.json, smallgraphs.txt, isgci_sage.xml), 336 kB; the SPKG.rst explains: The code used to parse the data from Andries E. Brouwer's website is available at https://github.com/nathanncohen/strongly_regular_graphs_database
• polytopes_db (various files), 41 kB
• polytopes_db_4d (various files), 878 MB

Prepared files:

• conway_polynomials (Upstream Python distribution package for conway_polynomials #32747, Update to new conway-polynomials python package #36765)
• elliptic_curves

New data sources:

• Interface to the KnotInfo and LinkInfo databases #30352 Interface to the KnotInfo and LinkInfo databases
• database_graver_ppi - https://www.math.ucdavis.edu/~mkoeppe/art/ppi/index.html
• https://mathdb.mathhub.info/
• https://swmath.org/

Related (other stuff written by the Sage distribution into $SAGE_SHARE):

• Meta-ticket: Use system Jupyter notebook / JupyterLab #30306 - jupyter notebook related things
• Meta-ticket: Sage docbuild #20080 - documentation
• sage.combinat.cluster_algebra_quiver: Use sage.feature to define database file system location #30885 - sage.features

Techniques:

• importlib.resources - would help making a distribution zip_safe (sage.repl: Replace use of SAGE_EXTCODE by importlib.resources #31306)
• could consider advertise install location using a pkgconfig file: #30787 comment:24
• Reduce local duplication of large database package when venvs are in use: https://pypi.org/project/pydupes/ (similar to rdfind)
• Avoid duplicate installation by using PEP 660 editable wheels. Use a build backend that supports it: flit, pdm, hatchling, poetry, not setuptools. https://discuss.python.org/t/pep-660-and-setuptools/14855
• However there are implementation restrictions in the interaction of editable wheels and implicit namespace packages

CC: @mkoeppe @jhpalmieri @kiwifb @isuruf @antonio-rojas @dimpase @slel @tobihan @jamesjer @soehms @videlec @williamstein @roed314

Component: build

Keywords: sd111

Issue created by migration from https://trac.sagemath.org/ticket/30914