We modularize sagelib into separate distributions (pip-installable packages), so that these can be built and run with a much smaller set of dependencies. The packages are made available individually on PyPI in source and wheel form. The modularization is done without changing the overall structure of the source tree by means of PEP 420 implicit namespace packages.

• SageMath modularization project: The five by five (sage-devel, 2023-06)
  I. The goals – II. The tools – III. The hooks – IV. The rules – V. The blocs
• Video from Sage Days 120
• Section Packaging the Sage Library in the Sage Developer's Guide (added in Sage 9.5)

Mega-PR #35095 (not for merging; changes are being cherry-picked from here for integration in Sage) adds the new distributions. Instead of defining a distribution package for every possible community or subfield of mathematics that Sage supports, the modularization project introduces three types of distribution packages:

Distributions named after a basic mathematical structure: The packages may also cover a wide range of generalizations/applications of the structure after which they are named. Users who work in a specialized research area will, of course, recognize what structures they need. The down-to-earth naming also creates discoverability by a broader audience. Not many more distribution packages than these 7 are needed:

• sagemath-combinat: provides "everything combinatorial", except for graphs.
• sagemath-graphs: also provides posets, combinatorial designs, abstract simplicial complexes, quivers, etc.
• sagemath-groups
• sagemath-modules: also has matrices, tensors, homology, coding theory, etc.
• sagemath-polyhedra: also provides fans, hyperplane arrangements, polyhedral complexes, linear and mixed-integer optimization, lattice point sets, and toric varieties
• sagemath-schemes
• sagemath-symbolics: symbolic expressions and symbolic calculus

Distributions named after a third-party non-Python dependency: This makes technical sense because the dependencies will be localized to this distribution package, but it also helps give attribution and visibility to these libraries and projects that Sage depends on.

Standard packages	Optional Packages
sagemath-flint	sagemath-bliss
sagemath-gap	sagemath-brial
sagemath-glpk	sagemath-coxeter3
sagemath-giac	sagemath-mcqd
sagemath-homfly	sagemath-meataxe
sagemath-lcalc	sagemath-sirocco
sagemath-libbraiding	sagemath-tdlib
sagemath-libecm
sagemath-linbox
sagemath-ntl
sagemath-pari
sagemath-singular

Distributions named after a technical functionality:

• sagemath-objects: Sage extends Python's object system by dynamic mix-in classes that are driven by categories and axioms. It is loosely modeled on concepts of category theory and inspired by Scratchpad/Axiom/FriCAS, Magma, and MuPAD. This distribution package makes Sage objects, the element/parent framework, basic categories and functors, the coercion system and the related metaclasses available.
• sagemath-categories: This distribution package contains the full set of categories defined by Sage, as well as basic mathematical objects such as integers and rational numbers, a basic implementation of polynomials, and affine spaces. None of this brings in additional dependencies.
• sagemath-environment
• sagemath-plot: Plotting facilities, depending on matplotlib.
• sagemath-repl: The top-level interactive environment with the preparser that defines the surface language of Sage. This distribution also includes the doctesting facilities, as the doctests are written in the surface language.
• sagemath-standard: Everything as provided by a standard installation of the Sage distribution. This is reduced to an empty meta-package.
• sagemath-standard-no-symbolics

---

Older posts and presentations:

• Initial proposal (sage-devel, 2020-05)
• Presentation video: https://researchseminars.org/talk/SageDays110/11/ (2020-10)
• Schematics (2020-10)
• Current status of pip-installability (Sage 9.4 release tour, 2021-08)
• Refactoring tasks in the Sage library to support modularization (sage-devel, 2021-09)
• How to modularize for fun and profit (sage-devel, 2021-10)
  • I. Namespace vs. distribution name
  • II. MONOREPO vs. MULTIREPO
  • III. Dependencies, in 4 special flavors
  • IV. Testing subset distributions in virtual envs
• Master plan of sage modularization (sage-devel, 2021-11)
• Presentation video: https://researchseminars.org/talk/SageDays112358/17/ (2022-06)

---

Initial steps: (2020)

1. In particular, create a package sagemath-objects that makes Sage objects, the element/parent framework, categories, the coercion system and the related metaclasses available. This package would, for example, not provide or know about integers, polynomials, or matrices (thus avoiding all the dependencies on the various libraries implementing these). Ideally it would only have a build dependency on Cython and no runtime dependencies.

   (Doctesting sagemath-objects would still need more of Sage, but some it could also be done by mocking instead of with the real classes.)

   a) These facilities could see a wider use in the mathematical Python community, and in this way we could hope to benefit from a larger developer base. For example, we would hope that sagemath-objects could be ported for use with PyPy instead of CPython.

   b) Building and deploying a user package would be easier if it could depend on a small package such as sagemath-objects only instead of the whole multigigabyteness of Sage.

   This has been completed in Sage 9.6 (Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865).

2. Remove the mechanism of OptionalExtensions from sagelib.

   Currently, a user would install, for example, the optional package tdlib and then rebuild sagelib so that the OptionalExtension sage.graphs.graph_decompositions.tdlib is built and installed.

   With this ticket, the user would instead install a new optional package sagemath-tdlib (which has tdlib as a dependency); this installs the Extension sage.graphs.graph_decompositions.tdlib (as a namespace package).

   This has been completed in Sage 9.2 (Replace use of module_list and OptionalExtension by extending find_python_sources #29701).

Not within the scope of this ticket:

• We will not change the overall structure of the source tree (SAGE_ROOT/src/sage). We will achieve this by using the mechanism of native namespace packages introduced in Python 3.3 (PEP 420, see Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420) #28925).

• It is not a prerequisite to fight against all circular imports. In particular, a parent class and its corresponding element class would certainly not be separated into separate distribution packages.

Initially noted constraints and challenges: (2020)

• Inconsistent handling of symlinks by pip/setuptools. We cannot simply create a distribution of a subset of the files in src by creating a flock of symlinks (see Install sage-env-config as part of sage_conf #29850) and expect everything to work; notably, pip fails because of longstanding issues with installs from local directories (see discussion in Install sage_conf using flit instead of setuptools #29854).

  Solution (implemented in Sage 9.2, Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels #29500): Do not use pip for installations from local source directories. Use setup.py bdist_wheel, followed by pip install of the wheel. pip 21.1's --use-feature=in-tree-build makes this workaround unnecessary (Use pip --use-feature=in-tree-build, replace use of sdh_setup_bdist_wheel by sdh_pip_install #32046, Sage 9.5).

• The category framework can not really be separated from the Parent/Element/CategoryObject (e.g. coercion is involved in comparisons and binary operations).

  Solution: Therefore they are in the same distribution sagemath-objects.

• The category framework cannot live without the integers (e.g. the output of .cardinality() expects an integer).

  Near term solution: Modularization of sagelib: Break out a separate package sagemath-polyhedra #32432 makes sagemath-categories larger, to include integers and some other basic rings.

  Possible future solution: The coercion system already has the notion of a global coercion model - which is determined at runtime, not compile time (sage.structure.element.get_coercion_model). When methods in sagemath-objects need to create an integer, they should obtain the appropriate class of integers in the same way. An existing function provides this interface already (but has a static implementation):

  sage: py_scalar_parent(int)
  Integer Ring
  

  sage.structure.coerce.py_scalar_parent, py_scalar_to_element: Make dynamic #29875 will make this implementation dynamic.

• Imports in sage.categories:

  bimodules imports QQ, RR
  classical_crystals imports SR
  finite_coxeter_groups imports QQbar
  

  Such imports can be removed using the techniques described in section Module-level runtime dependencies in the Sage developer's guide.

Tickets for Sage 9.2: (2020)

see also Modularization changes in Sage 9.2 (release tour)

• make sagelib a script package #29411
• Move all code from src/setup.py, src/fpickle_setup.py to sage_setup #29702
• Move Extension options from src/module_list.py to distutils: directives in the individual files: Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 1 - some packages without OptionalExtensions) #29706, Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 2 - OptionalExtensions) #29720, coxeter3: Add spkg-configure.m4, use coxeter/ prefix for includes, move OptionalExtension options to .pyx file directives #29721, Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 3: Get rid of uname_specific) #29785, Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 4: sage.rings) #29786, Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 5: sage.graphs) #29790, Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 6: last) #29791
• Replace use of module_list and OptionalExtension by extending find_python_sources #29701 (depends on all of the above)
• upgrade to ipython 7 #28197 (old ipython completer crashes on namespace packages)
• Upgrade setuptools, setuptools_scm, pip (2020-06), add package wheel; remove zope_interface #29803
• Install src/bin scripts by sagelib's setup.py, not make #21559
• Build sagelib from build/pkgs/sagelib/src, fix setup.py sdist, add spkg-src and tox.ini #29950 (supersedes: Fix sagelib sdist (src/setup.py sdist) #21516 / make sagelib use setuptools instead of distutils #13190)
• Improve/repair filtered structure of sagelib: Move attrcall and friends from sage.misc.misc to new module sage.misc.call #29869, sage.categories: Remove module-level imports of sage.rings, sage.algebras, sage.matrix, sage.misc.latex, etc. #29873, Move sage.misc.misc.coeff_repr, repr_lincomb to new module sage.misc.repr #29892, sage.structure: Remove some module-level imports #29883, Move RecursivelyEnumeratedSet_forest code to sage/sets/recursively_enumerated_set.pyx, remove deprecated classes SearchForest, TransitiveIdeal, TransitiveIdealGraded #16351, sage.categories: Replace imports from sage.rings.all by more specific imports #29881, sage.categories.pushout.type_to_parent: Reimplement using py_scalar_parent #29880, sage.structure, sage.misc: Replace import of sage.categories.all by more specific imports #29916, sage.doctest: Make imports more specific; make global environment for tests customizable #29922
• Move giacpy_sage into sage source code #29171
• Merge sage_brial into sagelib #30332
• Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels #29500
• sage.features.Feature.resolution: If SAGE_ROOT is available, recommend system packages #30606
• Replace use of sage.misc.package.PackageNotFoundError (2); lazy_import: Add keyword argument 'feature' #30616

Tickets for Sage 9.3: (2021)

see also Modularization changes in Sage 9.3 (release tour)

• Remove a few .all imports #30715
• Remove a few more .all imports #30739
• sage.matrix: Resolve circular imports without using __init__.py #30784
• Duplicate src/setup.py #30779
• sage_setup: Separate installing and cleaning #30780
• Clean up sage_setup.library_order #30579
• sage_setup: Remove import-time dependency (setup_requires) on pkgconfig, numpy #30580
• Sagelib's scripts in src/bin should not use build/bin/sage-system-python; remove sage-pypkg-location #29355
• sage_setup: Separate installing and cleaning #30780
• Add build/pkgs/SPKG/install-requires.txt for all Python packages, remove some unneeded packages #30719
• src/bin/sage-env: Make SAGE_ROOT and SAGE_LOCAL optional #29951
• sage: Determine SAGE_LOCAL, SAGE_ROOT via sage-config if it exists #29852
• Replace "$SAGE_LOCAL/bin" by more specific variables to make Sage easier to package, use in venvs #22731
• src/bin/sage-env: Make sure $SAGE_VENV/bin is at the beginning of the PATH #30013
• Remove import of 'ppl' at startup using lazy_import with feature #30587
• Update pplpy to 0.8.6 #31261
• Install sage-env-config as part of sage_conf #29850:
• FPLLL 5.4.0 and FPyLLL 0.5.4 #31025
• src/bin/sage-venv-config: Do not fail if sage_conf is not available #31058
• bootstrap: Fix "sage-venv-config: /doesnotexist/python3: bad interpreter" #31218
• Remove sage-location's "sage-force-relocate" mechanism, fix script to work without SAGE_ROOT #31270
• build/pkgs: Add some missing install-requires metadata, add some documentation to the developer's guide #31321
• sagemath-standard: sagelib: Update metadata for PyPI deployment as package sagemath-standard #30912 / Fixup src/VERSION.txt added in #30912 #31357
• sage.misc.persist: Import SAGE_DB on need only #31332
• sage.doctest.control: Do not fail if sage.libs.arb cannot be imported #31334
• sage_setup: Replace imports from distutils by setuptools #30984
• Add NTL to cython_aliases and sage.misc.cython library search dirs #31365
• Add package argon2-cffi (dependency of Jupyter notebook) #31279
• make build-local, make build-venv #31362
• ./configure --enable-editable #31377
• pip, wheel, setuptools: Make wheels available in SAGE_SPKG_WHEELS #31045
• sage-docbuild: Split sage_setup.docbuild out to a separate distribution sage_docbuild #30010
• widgetsnbextension should not depend on notebook #31278
• Add "configure --disable-notebook"; show descriptions of optional packages in "configure --help" #30383
• Fix dependencies of ipywidgets #31523
• Make it possible to disable build of r, rpy2 using ./configure --disable-r #31409

Tickets for Sage 9.4: (2021)

see also Modularization changes in Sage 9.4 (release tour)

• sagelib: Generate pyproject.toml and setup.cfg [install_requires], requirements.txt, Pipfile, and src/Pipfile #30913 (supersedes Add src/requirements.txt for installation of sagelib in a venv #30578, sagelib: Declare build system dependencies using src/pyproject.toml #30581)
• cython_aliases: Do not fail if one of the listed libraries is not known to pkgconfig #31384
• Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL #29013
• configure: Paths within $SAGE_LOCAL ($prefix) for sage_conf #31593
• sage_setup: Use paths within SAGE_LOCAL when provided via sage_conf #31338
• Patch giac so that libintl from /usr/local does not leak in on macOS #31562
• Install fewer static libraries #28890
• sage-env, sage.env: Remove unused environment and Python variables SAGE_ETC, SAGE_INC #32045
• sage.env: Remove direct uses of SAGE_LOCAL from the Sage library #32036
• sage_setup.setenv: Only set ...PATH variables and LDFLAGS if SAGE_LOCAL differs from SAGE_VENV #32057
• Move SAGE_ROOT/build/pkgs/*/src to SAGE_ROOT/pkgs/* #31577
• sage.env.sage_include_directories: Remove hard dependency on numpy #31333
• Remove sage.libs.readline #32073

Tickets for Sage 9.5: (2022)

see also Modularization changes in Sage 9.5 (release tour)

• Use pip --use-feature=in-tree-build, replace use of sdh_setup_bdist_wheel by sdh_pip_install #32046
• Install sage_setup with setuptools as a prerequisite for building/installing sagelib #29847. Deploy distribution sage-setup to PyPI.
• ptyprocess: Pin version to 0.5.1 in install-requires.txt #32371
• Change 'sage_setup: distribution' headers from sage-... to sagemath-... #31031
• Merge pynac sources as src/sage/symbolic/ginac #32386 (supersedes Repackage pynac as a pip-installable package #30534)
• Deprecate sage.interfaces.primecount #32412 ... and sage.libs.primecount
• sage_setup.library_order: Skip libraries that were not detected #32373
• Meta-ticket: Clear out __init__.py files in preparation for namespace packages #32501:
  • Review/remove monkey patching in src/sage/__init__.py: Remove some monkey patching in src/sage/__init__.py #31420 / Remove monkey patching of inspect.isfunction in sage.__init__ #32479 / Move load_ipython_extension from sage.__init__ to sage.repl.__init__ #32489
  • __init__.py in other packages: Remove useless 'file is not empty' comments from __init__.py files #32506 / Remove useless 'from . import all' from some __init__.py files #32507 / Eliminate use of __init__.py files for supplying package docstrings #32508 / Clear out __init__.py in sage.rings.polynomial #32509
  • Fix unpickle_global when register_unpickle_override appears in a module not loaded at startup #32840
• Meta-ticket: Replace is_Class functions, create abstract base classes to enable modularization #32414:
  • sage.rings.abc: sage.rings.abc #32566 / Deprecate is_RealField, is_ComplexField, is_RealDoubleField, is_ComplexDoubleField #32610 / sage.rings.abc.{Real,Complex}{Interval,Ball}Field; deprecate is_{Real,Complex}IntervalField #32612 / Add sage.rings.abc.{AlgebraicField,...,NumberField_quadratic}, deprecate is_AlgebraicField, ..., is_CyclotomicField, is_QuadraticField #32660 / sage.rings.abc: Add pAdicRing, pAdicField; deprecate is_pAdic... #32750
  • sage.arith: Use sage.rings.abc and move some module-level imports into methods #32600
  • Replace is_IntegerModRing by isinstance with new class sage.rings.abc.IntegerModRing #32606
  • Use sage.rings.abc.RealField, sage.rings.abc.ComplexField more #32742
• Remove .all imports: sage.geometry: Remove .all imports #32534 / Remove some .all imports #32591 / Replace imports of QQ, ZZ, GF, Integer from sage.rings.all by more specific imports #32620 / Remove .all import of infinity #32734 / Remove more .all imports #32733
• ./configure: Make --with-sage-venv=auto the default #32442
• GH Actions: Deploy sdists to PyPI #32062
• pip-installable version of package sage_conf - installs non-Python bits of the Sage distribution in ~/.sage/ #29039
• Modularization of sagelib: Break out a separate package sagemath-standard-no-symbolics #32601: <!--- Modularization of sagelib: Break out a separate package sagemath-standard-no-symbolics --->
  • Remove unnecessary uses of SR and symbolic functions: Remove unnecessary uses of SR and symbolic functions in sage.algebras, sage.combinat #32411 / Remove unnecessary uses of SR and symbolic functions in sage.categories, sage.coding #32413 / Remove unnecessary uses of symbolic functions in sage.tensor.modules #32415 / Remove unnecessary uses of SR and symbolic functions in sage.geometry (except .hyperbolic_space) #32416
  • sage.structure, sage.sets: Remove hard dependencies on the symbolic ring / polynomial rings #32599
  • sage.modules, sage.matrix: Remove import-time dependencies on the symbolic ring #32593
  • sage.structure.element: Add abc Expression; deprecate is_Expression etc. #32638
  • Deprecate is_[Callable]SymbolicExpressionRing, remove use of is_Symbolic{Equation,Variable}, is_CallableSymbolicExpression #32665
  • sage.tensor.modules: remove dependency on sage.manifolds #32708
  • sage.tensor.modules: make doctests involving SR optional #32712
  • Modularization changes in sage.modules, sage.matrix #32725
  • Import Expression from sage.structure.element #32730
  • Move sage.functions.other.sqrt to sage.misc.functional #32717
  • Avoid ceil, floor from sage.functions.other in non-symbolic uses #32731
• sage.doctest: Eliminate hard dependency on RealIntervalField #32585
• Features and optional tags for sage modules provided by separate distributions #32614
• sage.doctest.control: Exclude doctests in files via file directives ''# sage.doctest: optional - xyz' #30778
• Replace sage.doctest.external.has_* functions by Features #32649
• sage.misc.latex: Replace have_... functions by Features #32650
• lazy_import: Muffle diagnostic messages for "at_startup" #32619
• Modularization of sagelib: Break out a separate package sagemath-polyhedra #32432: <!--- Modularization of sagelib: Break out a separate package sagemath-polyhedra --->
  • Remove compile-time dependency on cypari2 and flint from some modules: sage.rings.integer, rational: Remove compile-time dependency on cypari2 and flint #30022 / sage.rings.fast_arith: Remove compile-time dependency on cypari2 #32441 / sage.matrix.args: Remove compile-time dependency on cypari2 #32455 / Mostly fix slow down by #30022 #32775 / sage.rings.{real,complex}_double: Remove compile time dependency on cypari2 #32701
  • sage.geometry.polyhedron: Split out backend_cdd_rdf from backend_cdd #32592
  • ConvexSet_base, Polyhedron_base: In _test methods, do not fail if some modules are unavailable #32628
  • sage.misc.latex, sage.repl.display.fancy_repr: Make imports more local #32634
  • sage.geometry.polyhedron: Mark doctests # optional - sage.rings.number_field etc.: sage.geometry.polyhedron: Mark doctests # optional - sage.rings.number_field #32652 / sage.geometry.polyhedron: Mark doctests # optional - sage.combinat #32653 / sage.geometry: More # optional for test with examples that need sage.combinat, sage.rings.number_field, sage.plot #32732
  • sage.matrix.matrix_space: Import element classes on demand, fall back to generic on ImportError #32635
  • Polyhedron_base.vertex_adjacency_matrix: Do not use face_lattice #32666
  • sage.rings.real_double: Move methods using GSL to a separate extension module #32677
  • Parent._is_numerical, _is_real_numerical - remove hard dependency on real_mpfr, complex_mpfr #32625
  • Decentralize sage.rings.numbers_abc #32641
  • sage.categories.coxeter_groups: Remove module-level imports from sage.rings, sage.graphs #32769
  • Allow some imports of RLF to fail #32777
  • sage.geometry.polyhedron.library: Delay import of rings #32780
  • sage.rings.infinity: Remove hard dependency on real_lazy and real_mpfi #32778
• Move list of scripts and package_data from setup.py to setup.cfg #31386
• Developer's guide: Add section on modularization #32899
• Split out sage.libs.primecount/sage.interfaces.primecount as a separate Python library: update and promote spkg primecount #25009 / split out a separate Python package primecountpy #32894:

Tickets for Sage 9.6: (2022)

see also Modularization changes in Sage 9.6 (release tour)

• Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865: These distributions only depend on Cython, pkgconfig, cysignals, gmpy2. Use them as a testsuite for testing the filtered structure of sagelib.
• Remove unused class from sage_setup.optional extension #31013
• sage.geometry: More # optional #32846
• Use lazy_import for module-level import from sage.plot (outside of sage.plot) #32847
• sage.repl: Replace use of SAGE_EXTCODE by importlib.resources #31306
• Features: Feature for sage.rings.padics #32881 / Feature for sage.groups #32888
• Prepare sage.doctest for namespace packages #33033
• Refactor {Matrix,Vector}_double_dense through ..._numpy_dense, add ..._numpy_integer_dense #32465
• sage.geometry.polyhedron: More # optional #33402
• Polyhedron._test_product, _test_dilation: Skip tests if test prereqs cannot be imported #33403
• sage.matrix: Modularization fixes after #33159 #33392 (wrong result solving equation system with symbolic matrix #33159)
• Remove imports from sage.*.all: Remove imports from sage.misc.all #32989 / Remove imports from sage.libs.all #32999 / Remove some imports from sage.rings.all #33000 / Remove imports from sage.interfaces.all #33007 / Replace imports from sage.*.all by more specific imports in combinat #33146 / Replace imports from sage.graphs.all by more specific imports #33199
• Remove or tag uses of SR and symbolic functions in combinat, categories, etc. #32609
• Features for palp, polytopes_db, polytopes_db_4d #32893
• sage.features.Executable: Prepend $SAGE_VENV/bin:$SAGE_LOCAL/bin to PATH #31296
• sage.libs.singular: Do not fail if the Singular binary is not in PATH #33440
• Use Executable.absolute_filename: sage.graphs: Use Executable.absolute_filename() #33465, sage.features.lrs: Make it a JoinFeature of Lrs, LrsNash; use Executable.absolute_filename #33466, sage.geometry.polyhedron.backend_cdd: Go through sage.features for _cdd_executable #32645, sage.geometry: Use PalpExecutable(...).absolute_filename() #33467
• LazyImport.__instancecheck__, __subclasscheck__: Return False on ImportError #33017

Tickets for Sage 9.7: (2022)

see also Modularization changes in Sage 9.7 (release tour)

• Remove some uses of SAGE_TMP: sage.misc.temporary_file: Remove use of SAGE_TMP #33797 / Replace SAGE_TMP in doctests of sage.misc.{persist,ostools}, sage.doctest, sage.repl #33799 (supersedes sage.misc.temporary_file: Move SAGE_TMP implementation here #32986)
• sage.misc.sage_input: Remove import from sage.all #33801
• Fixes for the distributions sagemath-objects, sagemath-categories #33803
• pkgs/sagemath-*/tox.ini: Create environment that uses Sage's python as the basepython, fix sagemath-standard deps #32716
• Modularization of sagelib: Break out separate packages sagemath-environment, sagemath-repl #29941
• Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420) #28925, rework sagemath-objects, sagemath-categories as namespace packages
• python3 spkg-configure.m4: Require ensurepip #33822 python3 spkg-configure.m4
• Move handling of "sage --docbuild" back to src/bin/sage #33795
• Remove use of SAGE_LIB in sage.misc #33821
• Refactor sage_conf #33295
• Feature and doctest tag for runtime cython #33029
• cibuildwheel workflow for sagemath-objects, sagemath-categories #33800
• Remove __init__.py files for subpackages designated to be namespace packages #33011
• Remove src/sage/__init__.py #34187
• Make "./configure --enable-editable" the default #32406
• GH Actions: Add test of the modularized distributions #33817
• Backport PEP420 namespace package support from Cython 3 #34221
• Remove .all imports: Remove imports from sage.all and sage.rings.all in sage.rings #34189 / Remove imports from sage.rings.all in sage.schemes #34190 / Remove imports from sage.rings.all in sage.modular #34191 / Remove imports from sage.rings.all in sage.calculus, functions, symbolic #34192 / sage.manifolds, sage.tensor: Remove imports from sage.arith.all, sage.rings.all #34428
• GH Actions: Upload wheels to PyPI #34296

Tickets for Sage 9.8: (2023)

see also Modularization changes in Sage 9.8 (release tour)

• Refactor distributions sagemath-{objects,categories} through sagemath-{environment,repl} #33812
• New option "./configure --enable-wheels" #32874 (supersedes build/pkgs/sagelib: Invoke install cleaner explicitly, remove it from pkgs/sagemath-standard/setup.py #32927, sage_setup: Modify clean_stale_files to support out-of-tree namespace packages #30152)
• Deprecate sage.interfaces is_...Element functions: fix coercion from libgap's finite fields, use libgap in sage/rings/finite_rings #34770/deprecate is_GapElement #34823/Deprecate sage.interfaces is_...Element functions #34804
• lazy_import doesn't resolve properly when indirectly imported #16522
• Interfaces: use more lazy imports, restore top-level functions maxima_console etc. #34547
• Fix sagemath-standard sdist #34858

Tickets for Sage 10.0 (2023)

• Refactoring tool to fix modularization anti-patterns #34945
• Remove .all imports: sage.{algebras,combinat,matroids}: Replace imports from sage.*.all for namespace packages #34946 / sage.{rings,modules,geometry}: Replace imports from sage.*.all for namespace packages #34947 / sage.{coding,groups}: Replace imports from sage.*.all for namespace packages #34948 / sage.{categories,matrix,structure}: Replace imports from sage.*.all for namespace packages #34949 / sage.schemes: Replace imports from sage.*.all for namespace packages #34951 / sage.graphs: Replace imports from sage.*.all for namespace packages #34952 / sage.{topology,homology}: Replace imports from sage.*.all for namespace packages #34953 / sage.{functions,interfaces,symbolic}: Replace imports from sage.*.all for namespace packages #34954 / sage.{arith,crypto,databases,dynamics,lfunctions,quadratic_forms}: Replace imports from sage.*.all for namespace packages #34955 / sage.{misc,monoids,sets}: Replace imports from sage.*.all for namespace packages #34956 / sage.{finance,interacts,libs,numerical,stats,tests}: Replace imports from sage.*.all for namespace packages #34957 / Meta-ticket: Replace imports from sage.*.all for namespace packages #34201 / Replace more .all imports #35372
• sage.geometry.integral_points: Use generic impl if no matrix_integer_dense #35135
• New ABC sage.structure.element.NumberFieldElement, deprecate is_NumberFieldElement #35100
• Meta-PR: More # optional doctest tags #34998
• Deprecate is_FiniteField etc., make sage.rings.finite_rings a namespace package #32664
• Add ABCs CommutativePolynomial, MPolynomial_libsingular, InfinitePolynomial; deprecate is_Polynomial, is_MPolynomial #32709
• ABC for BooleanPolynomialRing #35240
• Deprecate is_Algebra, is_CommutativeAlgebra #35253
• Many more namespace packages #35322 / Many more namespace packages – follow up #35366
• sage.features: Add sage.libs.singular, features for standard Python packages #35237
• Modularization fixes: Remove direct use of "setup.py sdist", add targets "make SPKG-sdist" #34855 / sage.matrix.operation_table: Modularization and code style fixes #35153 / sage.rings.polynomial.laurent_polynomial_ring_base: Split out from .laurent_polynomial_ring #35229 / sage.rings.function_field: Modularization fixes #35230 / sage.quadratic_forms: Remove module-level imports from number_fields, sage.symbolic, sage.functions #35243 / sage.topology: Move imports from sage.graphs, sage.homology into methods #35263 / sage.categories: Modularization fixes for imports #35279 / Modularization fixes for imports of number fields #35283 / sage.quadratic_forms: Modularization fixes for imports #35305 / sage.groups.matrix_gps: Modularization fixes for imports #35306 / sage.rings.finite_rings.residue_field: Modularization fixes #35389
• Add # optional: sage.{geometry,rings}: More # optional tags in doctests #35136 / sage.graphs: Add # optional doctest tags for modularization #35266 / sage.manifolds, sage.tensor: Add # optional doctest tags for modularization #35267 / sage.schemes: Reformat doctests, add # optional annotations #35314

Tickets for Sage 10.1 (2023)

• sage.rings: Reformat doctests, add # optional annotations #35457
• sage.rings.factorint: Modularization fixes #35502
• Laurent polynomial/series modularization fixes #35554
• sage.{topology,homology}: Modularization fixes #35581
• sage.geometry: Add some # optional, reformat doctests #35586
• sage.graphs: More # optional and other modularization fixes #35647
• sage.matrix.misc: Split by library dependency #35758
• Reduce dependency to rainbow in sage.graphs.graph_coloring #35780
• New feature annotations # optional - sage.schemes sage.modular sage.libs.flint etc. #35728
• doctester: Check for consistency of # optional annotations #35620
• sage.misc.misc, sage.combinat: Modularization fixes #35564
• Refactor sage.matrix.misc #32433
• Remove abuse of predefined x in doctests #32721
• sage.graphs: More modularization #35718
• sage.matroids: Modularization fixes #35719
• sage.coding: Modularization fixes, doctest cosmetics, add # optional #35729
• Reference manual: Show modularized sagelib packages separately #35734
• sage.combinat: Split some Cython modules (modularization fixes) #35741
• Doctest # optional / # long annotation dataflow checker #35401
• New modularized distributions sagemath-{sirocco,bliss,meataxe,mcqd,coxeter3,tdlib} #34346
• sage.calculus: Modularization fixes, doctest cosmetics, # needs #35717
• sage -fixdoctests: Handle directory names, call sage -t only once #36024
• sage.graphs: Update # needs, use block-scoped tags #36026
• sage.groups.perm_gps.partn_ref*: Modularization fixes #35881
• New modularized distributions sagemath-{sirocco,bliss,meataxe,mcqd,coxeter3,tdlib} #35661
• sage.matroids: Update # needs, modularization fixes for imports #35919
• Integer.{perfect_power,is_prime_power,is_irreducible}: Handle easy cases without PARI #35847
• sage.combinat.designs: Modularization fixes, update # needs #35943
• sage.combinat.cluster_algebra_quiver: Modularization fixes, update # needs #35951
• sage.rings.function_field: Update # needs #35957
• pkgs/sagemath-meataxe/README.rst: Remove markup unknown to PyPI #36003
• CI Build & Test: Fix test errors involving optional packages coxeter3, ... #36016
• pkgs/sagemath*: Exclude all__*.py files of other distributions #36015
• Add style guide / reference for # optional - sage.... doctest tags, extend sage -t and sage -fixdoctests for modularization tasks #35749
• sage.{modular,schemes}: Modularization fixes for imports; update sage -fiximports, add relint pattern  #35884
• sage.rings.{padics,valuation}: Modularization fixes, # needs #36052
• sage.geometry: Update # needs, use block-scoped tags #36033
• sage.graphs: fix doctest warnings due to modularization #36077
• sage.typeset: Update # needs #36037
• sage.rings.polynomial: Modularization fixes, # needs #36045
• sage.rings.number_field: Modularization fixes, doctest cosmetics, # needs #36044
• sage.plot: Add/update # needs #36038
• sage.rings.finite_rings: Modularization fixes, # needs #36056

Tickets for Sage 10.2 (2023)

• using # needs in posets folder #36114
• sage.crypto: Update # needs, modularization fixes #36106
• sage.geometry.polyhedron: fix doctest error due to modularization #36089
• sage.{cpython,data_structures,databases,ext,parallel,structure}: Update # needs #36067
• sage.arith: Update # needs #36057
• sage.categories: Update # needs #35945
• sage.functions: Decouple from sage.symbolic.expression and mpmath, update # needs #35716
• sage.rings: Update # needs #36152
• sage.schemes: Update # needs #36102
• sage.groups: Modularization fixes, # needs, doctest cosmetics #36168
• sage.rings: Remove code deprecated in #23204, #24483, #24371, #24511, #25848, #26105, #28481, #29010, #29412, #30332, #30372, #31345, #32375, #32606, #32610, #32612, #32641, #32660, #32750, #32869, #33602 #36307
• sage --fixdoctests --update-known-test-failures; silence modularized distributions in CI #36264
• sage.sets: Update # needs #36272
• make sagemath_categories-check, make pypi-wheels-check #36452
• sage.symbolics: Remove code deprecated in #18036, #29738, #32386, #32638, #32665, #34215 #36304
• pkgs/sagemath-{bliss,sirocco,tdlib}: Fix sdists #36520
• Conditional documentation #36495
• GH Actions: Fix wheel build #36521
• pkgs/sagemath-standard: Fix sdist #36532
• sage.{dynamics,schemes}: Modularization fixes, docstring cosmetics, update # needs #36271
• sage.rings.padics: Update # needs #36259
• pkgs/sagemath-{objects,categories,environment,repl}: Move metadata from setup.cfg to pyproject.toml #36563
• sage.numerical: Update # needs #36567
• sage.manifolds, sage.tensor: Update # needs #36568
• sage.matrix, sage.modules: Update # needs #36569
• Replace relative imports by absolute ones in sage.{matrix,matroids,modules,stats} #36594
• Replace relative imports by absolute ones in sage.{coding,combinat,graphs} #36595
• Replace relative imports by absolute ones in sage.{geometry,groups,numerical,plot} #36596
• Relativize header imports #36598
• Replace relative imports by absolute ones in sage.libs.ntl #36605
• Replace relative imports by absolute ones in sage.libs except .ntl #36612
• sage.modular: Update # needs, doctest cosmetics #36618
• sage.combinat: Update # needs #36619

Tickets for Sage 10.3 (2023–2024)

• sage.rings: Modularization fixes #36917
• src/sage/{categories,coding,plot,quadratic_forms}/: Fix various doctest warnings #36905
• src/sage/game_theory/: fix doctest warnings due to modularization #36904
• sage.plot: Update # needs, doctest cosmetics; fix sig_on_count doctest dataflow warnings #36950
• build/pkgs/gnumake_tokenpool: Update to fix AlarmInterrupt problems #36948
• sage.groups, sage.rings.number_field: Modularization fixes, # needs #36865
• Fix build of optional Cython modules broken in #29386 #36900
• Use GNU make tokenpool protocol to manage parallelism of doctesting #36640
• build/bin/sage-spkg: Add support for installing script packages #36747
• src/sage/libs/pari/convert_sage_matrix.pyx: Remove sage_setup: distribution #36859
• pkgs/sagemath-{bliss,coxeter3,...}: Move metadata from setup.cfg to pyproject.toml #36564
• sage.libs.pari, sage.rings.real_mpfr: Modularization fixes #36645
• sage.interfaces: Update # needs #36656
• Replace relative imports by absolute ones in sage.{algebras,arith,categories,cpython,data_structures,misc,modular,rings,sat,symbolic} #36666
• src/doc: Update # needs #36679
• build/bin/sage-spkg: Add options for checking, cleaning separately from build/install #36738
• Normal Python packages: postpone wheel installation to the post-install phase #36743
• sage -fixdistributions #36135
• sage.algebras: Update # needs, modularization fixes, doctest cosmetics #36642
• pkgs/sage-{docbuild,setup,sws2rst}: Migrate from setup.cfg to pyproject.toml #36562
• Replace relative imports by absolute ones in categories #36572
• Replace relative imports by absolute ones in a few packages #36589
• Replace relative imports by absolute ones in modules #36597
• sage.algebras: Update # needs, modularization fixes, doctest cosmetics #36642
• sage.combinat.species: Update # needs #36643
• sage.combinat.words: Update # needs #36644
• sage.tests: Update # needs #36657
• sage.sat: Update # needs #36658
• FriCAS spkg-configure and Feature #35838/Feature for fricas #33575

Tickets for Sage 10.4 (2024)

See also: Modularization changes in Sage 10.4 (release notes)

• HTML documentation: Show preparsed doctests using inline tabs #37083
• pkgs/sagemath-repl/pyproject.toml.m4: Declare extra 'sphinx' #37720
• Build system: make SPKG-check for normal and script packages #37022 / Use pypa/build instead of pip wheel #35618
• Restructure sage.*.all for modularization, replace relative by absolute imports #36676 / pkgs/sagemath-standard: Move metadata from setup.cfg to pyproject.toml #36951 / Add # sage_setup: distribution directives to all files, remove remaining # coding: utf-8 #36964 / Revert PR with disputed dependencies #37796 / Add # sage_setup: distribution directives for the files of existing distributions sagemath-{categories,...,tdlib} #38088 / all*.py files: Use 'del lazy_import', 'del install_doc' #38086
• CI: CI build, doc-build: Run containers explicitly, separate jobs for pyright, build, modularized tests, long tests #36498 / CI Build&Test: Fix "test modularized distributions" after #37022 #37750
• sagemath-{objects,categories} pyproject.toml: Add 'external' section according to draft PEP 725 #37486
• Replace special handling of optional extensions (bliss, coxeter3, ....) #37857
• sage.rings: Modularization fixes for singular #38060
• sage.{calculus,functions,numerical,symbolic}: Docstring/doctest cosmetics, # needs #37715 / sage.categories: Update # needs, use block tags #38071 / sage.coding: Update # needs #38104 / src/sage/categories/hecke_modules.py: Fix # needs #38146
• Import fixes: sage.rings: Modularization fixes (imports) #38061 / sage.geometry: Modularization fixes (imports), # needs #38066 / sage.combinat: Modularization fixes (imports), update # needs #38074 / sage.monoids: Modularization fixes, # needs #38143 / sage.rings: Modularization fixes (imports), # needs #38210 / sage.algebras...sage.topology: Doctest cosmetics, # needs, import fixes #38193 / sage.categories: Update # needs, modularization fixes (imports) #38170 / sage.plot, sage.repl: Modularization fixes (imports), # needs #38181
• sage.interfaces: More lazy_import #38180 / sage.combinat, sage.games: Use more lazy_import in all.py files #38096 / src/sage/algebras/steenrod/all.py: Use lazy_import, remove deprecated global import #38095
• Deprecations: deprecated functions is_Cone, is_Polyhedron, is_LatticePolytope #37057 / sage.modular: Deprecate is_... functions #38035 / sage.combinat.finite_state_machine: Deprecate is_... functions #38032 / Deprecate is_Category, is_Endset, is_Homset, is_Parent, is_RingHomset, is_SimplicialComplexHomset #37922 / Deprecate is_FGP_Module, is_FilteredVectorSpace, is_FreeQuadraticModule, is_FreeModule, is_FreeModuleHomspace, is_MatrixSpace, is_Module, is_VectorSpace, is_VectorSpaceHomspace #37924 / sage.modular.hecke: Deprecate is_Hecke... functions #37895 / Deprecate is_FreeAlgebra, is_QuaternionAlgebra, is_SymmetricFunctionAlgebra #37896 / Deprecate is_AbelianGroup, is_DualAbelianGroup, is_MatrixGroup #37898 / Deprecate is_AlgebraicNumber, is_AlgebraicReal, is_ComplexDoubleElement, is_ComplexIntervalFieldElement, is_ComplexNumber, is_FractionField, is_FractionFieldElement, is_Integer, is_IntegerMod, is_IntegerRing, is_Rational, is_RationalField, is_RealDoubleElement, is_RealIntervalField, is_RealIntervalFieldElement, is_RealNumber #38128 / Deprecate is_FreeAlgebraQuotientElement, ... #38184
• GH Actions: Fix build of macOS x86_64 wheels, build wheels for sagemath-{bliss,coxeter3,mcqd,tdlib} for PyPI #36525
• sage.libs.pynac: Remove, deprecated in #32386 (2021) #37873 / Fix sage.symbolic feature after libs/pynac removal #38081
• sage.calculus.expr: Split out from sage.calculus.all #37993
• GH Actions: Build platform-independent wheels of sagemath-environment, sage-setup, sage-sws2rst for PyPI #37099
• Features: Activate/fix rpy2 tests #35396 / sage.plot.animate: Use FFmpeg.absolute_filename; sage.features: Remove deprecated methods #37312
• build/pkgs/sagemath_{categories,environment,objects,repl}: Change to optional #38200

Tickets for Sage 10.5 (2024)

See also: Modularization changes in Sage 10.5

• Import fixes: sage.schemes: Modularization fixes (imports) #38142, sage.knots: Modularization fixes (imports), # needs #38118
• Deprecations: Deprecate is_Ideal, is_LaurentSeries, is_MPolynomialIdeal, is_MPolynomialRing, is_MPowerSeries, is_PolynomialQuotientRing, is_PolynomialRing, is_PolynomialSequence, is_PowerSeries, is_QuotientRing #38266, is_NumberFieldElement: Fix deprecation message #38286, Deprecate global imports: GroupExp_Class, GroupExpElement, GroupSemidirectProductElement #38238, Deprecate is_Infinite #38278, Deprecate is_SymmetricFunction #38279, Deprecate is_StringMonoidElement, ... #38280, Deprecate is_Ring #38288, Deprecate is_LaurentSeriesRing, is_MPowerSeriesRing, is_PowerSeriesRing #38290, Deprecate is_SchemeMorphism, is_SchemeTopologicalPoint #38296
• # needs: sage.geometry.hyperbolic_space: Add # needs #38236
• Features: sage.features.topcom #37858, sage.features.sat #38239
• src/pyproject.toml: Add 'external' section per draft PEP 725 (unbundled from #37446) #37482
• Meta-ticket: Remove runtime dependency on everything via sage_eval #33802
  • get rid of one sage_eval in complex numbers #38391
  • get rid of sage_eval in grobner_fan #38392
  • get rid of one sage_eval in modular #38394
  • get rid of some sage-eval in gap3-related code #38395
• Demote brial (= polybori) from standard to optional, add distribution sagemath-brial, enlarge sagemath-objects, sagemath-categories #36380

Tickets for Sage 10.x (2024)

• Mega-PR: Modularization of sagelib: Distributions sagemath-{brial,combinat,eclib,flint,gap,giac,glpk,graphs,groups,homfly,lcalc,libbraiding,libecm,linbox,modules,mpmath,ntl,pari,plot,polyhedra,schemes,singular,standard-no-symbolics,symbolics} #35095
• Define feature sage.libs.giac #38715, src/sage/features/giac.py: add new feature for the giac program #38672, Support for --disable-giac #38668, downgrade giac to optional #38712
• Restructure sage.*.all for modularization, replace relative by absolute imports #37900
• Add # sage_setup: distribution directives to all files #37901
• pkgs/sagemath-standard: Move metadata from setup.cfg to pyproject.toml #37902
• Distributions sagemath-{flint-arb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules #30666
• Migrate from setup.cfg to pyproject.toml #33577
• Modularization of sagelib: Break out separate package sagemath-linbox #31444
• Meta-ticket: Remove use of SAGE_LIB and SAGE_EXTCODE variables #33037
• Modularization of sagelib: Break out a separate package sagemath-polyhedra #32432
• relocatable wheel version of package sage_conf #31396
• Modularization of sagelib: Break out a separate package sagemath-standard-no-symbolics #32601
  • Remove module-level imports from sage.symbolic etc. (outside of sage.symbolic, sage.calculus etc.) #32718
• Prepare use of sage.all in sage.structure.factory.lookup_global for modularized distributions #32586
• Use src/setup.py for the installation mode --disable-editable --disable-wheels #34633
• Use PEP 660 editable wheels for sagelib #34209
• sage_setup: Output depfiles #34631
• Refactor sagemath-standard through sagemath-{categories,environment,repl} #34587
• eager_import #33820
• Merge cypari, cypari2; replace cysignal's optional PARI dependency by a general hook #32360
  • cysignals: Replace compile-time dependency on pari by run-time dispatch #32431
• Move sage.structure.formal_sum to sage.modules #29870
• sage.doctest.control: Replace use of sage.misc.package.list_packages #30746
• sage.doctest.control, sage_setup: Do not check versions of installed packages #30848
• src/tox.ini: Add "develop" environment #30911
• possibly move pkgs/sage-conf/sage_conf.py[.in] into src/, mark it (and also src/bin/sage-env-config[.in]) # sage_setup: distribution = sage-conf.
• pip-installable packages sagemath-doc-src, sagemath-doc-inventory, sagemath-doc-html, sagemath-doc-pdf #29868
• Meta-ticket: Eliminate use of patches for Python packages #31543
• Make lazy_import more friendly to pyflakes and other static checkers #30647
• Eliminate direct use of os.getenv in sage.interfaces #33405
• sage-conf_pypi: API for incremental builds #30036
• sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}" #29041
• Modularization of sagelib: Distribution sagemath-gsl #32700
• Towards making Arb library optional #30716
• Modularization of sagelib: Break out separate package sagemath-combinat #33660
• Modularization of sagelib: Distribution sagemath-plot #32702
• Modularization of sagelib: Break out a separate package sagemath-ntl #29912
• Modularization of sagelib: Break out separate package sagemath-symbolics #31695
• pynac: Replace direct use of singular's libfactory by a Python call #32613
• Eliminate use of FLINT's NTL interface #32908
• Use pipenv to create venv for development #30371

Wishlist tickets, loose ends

• Create PEP 503 simple repository for wheels built during installation #30527
• sage.categories: Replace import of ZZ by py_scalar_parent(int) #29877
• sage.structure.coerce.py_scalar_parent, py_scalar_to_element: Make dynamic #29875
• Modularization of sagelib: Mock integer class #29874
• sage_setup: Add PEP 517 / PEP 660 buildapi #29845
• modularized relocatable wheel version of sage_conf #31602
• Convert startup methods to context manager #30748
• ./configure --with-sage-venv=none #30896

Related:

• #21508
• https://wiki.sagemath.org/CodeSharingWorkflow
• #30914
• #30922
• #30818
  • #31041
• #31251
• #33580
• Meta-ticket: Replace imports from sage.*.all for namespace packages #34201

CC: @videlec @orlitzky @kiwifb @isuruf @tscrim @kliem @jhpalmieri @tobiasdiez @defeo @thierry-FreeBSD @anneschilling

Component: refactoring

Keywords: sd109, sd110, sd111

Issue created by migration from https://trac.sagemath.org/ticket/29705