-
-
Notifications
You must be signed in to change notification settings - Fork 655
Description
Is there an existing issue for this?
- I have searched the existing issues for a bug report that matches the one I want to file, without success.
Problem Description
Currently a few doctests use random input (e.g. random_element()
), which leads to unpredictable doctest failures (since depending on the input the tests might fail) and unreliable code coverage metrics (since depending on the input different code paths are executed).
Proposed Solution
Replace random input by deterministic/static input. If a test requires many different inputs to be convincing, it can be refactored using pytest parameterized tests.
Alternatives Considered
Accepts random test failures and unreliable coverage metrics.
Additional Information
A good starting point to see which tests are particularly affected is to go to https://app.codecov.io/gh/sagemath/sage/pulls, pick a PR at random, and look at the "Indirect changes" tab. These are changes in the coverage metrics that (most likely) are not coming from changes in the PR. For example, comparing https://app.codecov.io/gh/sagemath/sage/pull/35521/indirect-changes and https://app.codecov.io/gh/sagemath/sage/pull/35507/indirect-changes indicates that the tests in the following files are prone to lead different coverage:
- src/sage/modular/modsym/boundary.py
- src/sage/quadratic_forms/ternary_qf.py
- src/sage/rings/qqbar.py
- src/sage/combinat/posets/lattices.py
- src/sage/rings/polynomial/polynomial_ring.py
- src/sage/schemes/elliptic_curves/ell_generic.py
- src/sage/homology/chain_complex.py
- src/sage/numerical/interactive_simplex_method.py
and a few more. This can then be compared to the codecov report for the develop branch. For example, this line https://app.codecov.io/gh/sagemath/sage/blob/develop/src/sage/modular/modsym/boundary.py#L1385 is not covered in the develop branch but covered in the the PRs.