Add style guide / reference for `# optional - sage....` doctest tags, extend `sage -t` and `sage -fixdoctests` for modularization tasks #35749

mkoeppe · 2023-06-09T16:40:57Z

📚 Description

Fixing the handling of file-level # optional tags.
Some files were not being doctested; fixing the recently introduced errors in doctests.
Implementing block-scoped tags (originally done in PR Implement persistent optional tag #35779, merged here)
Expanding the documentation on the # optional / # needs tags used for modularization purposes, with examples.
Adding features sage.modular, sage.numerical.mip, sage.rings.complex_double, sage.sat
The tools sage -t and sage --fixdoctests receive some new options for modularization tasks (see added documentation):
```
$ make pypi-wheels
$ make SAGE_CHECK=yes sagemath-modules
$ ./sage --fixdoctests --distribution sagemath-modules \
    src/sage/combinat/root_system/root_lattice_realization_algebras.py
```
(example uses Modularization of sagelib: Distributions sagemath-{brial,combinat,eclib,flint,gap,giac,glpk,graphs,groups,homfly,lcalc,libbraiding,libecm,linbox,modules,mpmath,ntl,pari,plot,polyhedra,schemes,singular,standard-no-symbolics,symbolics} #35095)
Suppressing # optional/# needs of present features in the help system
.

Motivated by the sage-devel thread https://groups.google.com/g/sage-devel/c/utA0N1it0Eo (2023-06)

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

Depends on sage.features: Declare features as "standard" explicitly #35820 (merged here)
Vote at https://groups.google.com/g/sage-devel/c/8KZNRBs6F6U (result: https://groups.google.com/g/sage-devel/c/MtS2u3VbJEo)

…agemath-...

…g doctests, # optional

mkoeppe · 2023-06-16T14:52:31Z

(the doctest failure in ell_rational_field is unrelated)

gh-35734: Reference manual: Show modularized sagelib packages separately    ### 📚 Description  Packages such as **sagemath-categories** are currently mixed with optional SPKGs in the reference guide. We separate them out as a separate category. We also move the section on `Feature`s right next to the package list. #35749 and follow-ups will interlink these sections. Preview: - https://deploy-preview-35734--sagemath- tobias.netlify.app/reference/spkg/index.html#packages-of-the- modularized-sage-library - https://deploy-preview-35734--sagemath- tobias.netlify.app/reference/spkg/index.html#runtime-detectable- features-and-conditional-doctests   - Part of #29705  ### 📝 Checklist  - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: #35734 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

SageMath version 10.1.beta4, Release Date: 2023-06-21

…ks; add # optional on ModuleNotFoundError and 'was only set only in doctest marked' warning

kwankyu · 2023-07-12T01:02:49Z

I get

$ sage --fixdoctests --help
usage: sage-fixdoctests [-h] [-l] [--distribution DISTRIBUTION] [--venv VENV] [--environment ENVIRONMENT] [--no-test] [--full-tracebacks] [--only-tags]
                        [--probe FEATURES] [--keep-both] [--overwrite] [--no-overwrite]
                        [filename ...]

Given an input file with doctests, this creates a modified file that passes the doctests (modulo any raised exceptions). By default, the input file is
modified. You can also name an output file.

positional arguments:
  filename              input filenames; or INPUT_FILENAME OUTPUT_FILENAME if exactly two filenames are given and neither --overwrite nor --no-overwrite is
                        present

options:
  -h, --help            show this help message and exit
  -l, --long
  --distribution DISTRIBUTION
                        distribution package to test, e.g., 'sagemath-graphs', 'sagemath-combinat[modules]'; sets defaults for --venv and --environment
  --venv VENV           directory name of a venv where 'sage -t' is to be run
  --environment ENVIRONMENT
                        name of a module that provides the global environment for tests; implies --keep-both and --full-tracebacks
  --no-test             do not run the doctester, only rewrite '# optional/needs' tags; implies --only-tags
  --full-tracebacks     include full tracebacks rather than '...'
  --only-tags           only add '# optional/needs' tags where needed, ignore other failures
  --probe FEATURES      check whether '# optional - FEATURES' tags are still needed, remove these
  --keep-both           do not replace test results; duplicate the test instead and mark both copies # optional
  --overwrite           never interpret a second filename as OUTPUT; overwrite the source files
  --no-overwrite        never interpret a second filename as OUTPUT; output goes to files named INPUT.fixed

You may want to update the "positional arguments:" section.

kwankyu · 2023-07-12T01:54:12Z

src/sage/doctest/parsing.py

+
+        sage: print_with_ruler([
+        ....:     update_optional_tags('    sage: unforced()       # opt' 'ional - latte_int'),
+        ....:     update_optional_tags('    sage: unforced()  # opt' 'ional - latte_int',


Splitting the string containing # optional is still necessary? The doctester or the fixer is not robust enough yet to handle the case?

It is tox -e relint that is not smart enough

kwankyu

It looks very good to me as far as I can see.

kwankyu · 2023-07-12T02:21:21Z

As this PR is motivated by the sage-devel discussion started by @tscrim, I invite him to review this PR once more.

If there's no comment from him for a couple of days, I will set it positive review.

mkoeppe · 2023-07-14T16:08:53Z

Thank you!

vbraun · 2023-07-19T08:57:24Z

Doctests fail because now the previously-skipped broken files are picked up

sage -t --long --random-seed=286735480429121101562228604801325644303 sage/schemes/elliptic_curves/heegner.py  # 4 doctests failed
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/schemes/elliptic_curves/hom_frobenius.py  # 2 doctests failed

kwankyu · 2023-07-19T09:12:18Z

From sage-release

That's already fixed in #35749, which is why that PR is marked critical.

We didn't fix the failures. Just that @mkoeppe mentioned that they are not related with this PR. According to @vbraun, indeed they are.

kwankyu · 2023-07-19T10:25:02Z

They all look easy to fix. All the doctests rely on arbitrary choice between $P$ and $-P$ which are points of an elliptic curve whose $x$-coordinates are the same. I am 100% sure about the doctests in age/schemes/elliptic_curves/hom_frobenius.py and 80% sure about the doctests in sage/schemes/elliptic_curves/heegner.py. Needs a look from an expert @yyyyx4.

kwankyu · 2023-07-19T12:50:15Z

We may fix them in the style

--- a/src/sage/schemes/elliptic_curves/heegner.py
+++ b/src/sage/schemes/elliptic_curves/heegner.py
@@ -47,8 +47,8 @@ We find some Mordell-Weil generators in the rank 1 case using Heegner points::
     sage: E = EllipticCurve('43a'); P = E.heegner_point(-7)
     sage: P.x_poly_exact()
     x
-    sage: P.point_exact()
-    (0 : 0 : 1)
+    sage: p = P.point_exact(); p == E(0,0,1) or -p == E(0,0,1)
+    True
 
     sage: E = EllipticCurve('997a')
     sage: E.rank()

kwankyu · 2023-07-19T13:42:26Z

Please merge #35966 here for test.

github-actions · 2023-07-19T17:30:33Z

Documentation preview for this PR (built with commit 44549f6; changes) is ready! 🎉

gh-35919: `sage.matroids`: Update `# needs`, modularization fixes for imports    ### 📚 Description  - Many `# optional` are removed because small prime finite fields no longer need to be marked `# optional/needs sage.rings.finite_rings` since #35847 - Other `# optional` are replaced by codeblock-scoped `# needs` tags - Remaining `# optional` with standard features are replaced by `# needs` - Some import fixes needed for separate testing of **sagemath-modules** or **sagemath-graphs**[modules] from #35095   This is: - done with the help of `sage -fixdoctests` from #35749 - Part of: #29705 - Cherry-picked from: #35095  ### 📝 Checklist  - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: #35919 Reported by: Matthias Köppe Reviewer(s): David Coudert, Matthias Köppe

gh-35951: `sage.combinat.cluster_algebra_quiver`: Modularization fixes, update `# needs`   - `# optional` with standard features are replaced by `# needs` - Some import fixes needed for separate testing of **sagemath-graphs** from #35095   This is: - done with the help of `sage -fixdoctests` from #35749 - Part of: #29705 - Cherry-picked from: #35095 ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: #35951 Reported by: Matthias Köppe Reviewer(s): David Coudert

gh-35943: `sage.combinat.designs`: Modularization fixes, update `# needs`   - Many `# optional` are removed because small prime finite fields no longer need to be marked `# optional/needs sage.rings.finite_rings` since #35847 - Other `# optional` are replaced by codeblock-scoped `# needs` tags - Remaining `# optional` with standard features are replaced by `# needs` - Some import fixes needed for separate testing of **sagemath-graphs** from #35095   This is: - done with the help of `sage -fixdoctests` from #35749 - Part of: #29705 - Cherry-picked from: #35095    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: #35943 Reported by: Matthias Köppe Reviewer(s): David Coudert, Matthias Köppe

gh-35957: `sage.rings.function_field`: Update `# needs`   - Many `# optional` are removed because small prime finite fields no longer need to be marked `# optional/needs sage.rings.finite_rings` since #35847 - Other `# optional` are replaced by codeblock-scoped `# needs` tags - Remaining `# optional` with standard features are replaced by `# needs` - Some import fixes needed for separate testing of **sagemath-graphs** from #35095   This is: - done with the help of `sage -fixdoctests` from #35749 - Part of: #29705 - Cherry-picked from: #35095 ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: #35957 Reported by: Matthias Köppe Reviewer(s): David Coudert, Matthias Köppe

Matthias Koeppe and others added 13 commits June 9, 2023 09:35

src/sage/features/sagemath.py: Expand documentation on # optional - s…

664c56c

…agemath-...

src/doc/en/developer/coding_basics.rst: Add style guide for formattin…

d7ecd4d

…g doctests, # optional

src/sage/features/sagemath.py: Fix markup, expand examples

e7173cd

src/sage/features/sagemath.py: Add module-level doc

3aacb73

Fix markup

bfb9a52

src/sage/features/sagemath.py: Expand on sage.graphs

4b69756

Expand on sage.graphs, sage.modules

d46fb19

src/doc/en/developer/coding_basics.rst: Fix doctest output

5dc088f

Fix markup

1ad2776

src/doc/en/developer/coding_basics.rst: Fix doctest

eef016f

Merge branch 'develop' into feature_definitions

74b98cf

Add persistent optional tag

2440c0b

Fix a typo

647380c

kwankyu added 6 commits June 17, 2023 09:39

Allow needs for optional

2bd55ab

Block ends with a blank line

9485097

Use needs in the doctest

6dd4afd

Remove redundant backslash

f7152bd

Use formal needs:

87f5e15

Revert to needs revising the doctest

bdc4232

Matthias Koeppe added 9 commits June 23, 2023 14:21

Merge tag '10.1.beta4' into feature_definitions

0596705

SageMath version 10.1.beta4, Release Date: 2023-06-21

sage -fixdoctests: New options --venv, --environment, --full-tracebac…

131e1f9

…ks; add # optional on ModuleNotFoundError and 'was only set only in doctest marked' warning

sage -fixdoctests: Add option --keep-both

d9d1900

sage -fixdoctests: Handle why/explain annotation

dda8c28

sage -fixdoctests: Shorter annotations for top-level NameError

e9f38c0

sage -fixdoctests: Add option --only-tags

9541219

sage --fixdoctests: New option --distribution, defaults

6e25da2

sage --fixdoctests: Use git --no-pager

a14580e

sage --fixdoctests: Rewrite # optional properly

48b1660

Matthias Koeppe added 3 commits July 11, 2023 16:59

src/sage/doctest/parsing.py: Highlight keywords from relint

b801474

sage -t, sage -fixdoctests: Rename option --only-lib to --if-installed

fafaaf2

src/doc/en/developer/doctesting.rst: Standardize prompts

8d25508

./sage -fixdoctests: Update --help

0c42086

kwankyu reviewed Jul 12, 2023

View reviewed changes

kwankyu approved these changes Jul 12, 2023

View reviewed changes

kwankyu added s: positive review and removed s: needs review labels Jul 14, 2023

jhpalmieri added the c: doctest framework label Jul 19, 2023

vbraun added the s: needs work label Jul 19, 2023

Fix random doctests in elliptic curves

6b5b82c

kwankyu mentioned this pull request Jul 19, 2023

Fix random doctest failures in elliptic curves #35966

Closed

5 tasks

Merge branch 'fix-doctests-elliptic-curves' into feature_definitions

44549f6

tornaria mentioned this pull request Jul 21, 2023

Move hilbert_numerator test to its proper place #35977

Merged

jhpalmieri mentioned this pull request Jul 27, 2023

Deprecate sorting by default in connected component methods for graphs #35891

Merged

5 tasks

vbraun merged commit f2abd5a into sagemath:develop Jul 30, 2023

github-actions bot removed the s: positive review label Jul 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add style guide / reference for `# optional - sage....` doctest tags, extend `sage -t` and `sage -fixdoctests` for modularization tasks #35749

Add style guide / reference for `# optional - sage....` doctest tags, extend `sage -t` and `sage -fixdoctests` for modularization tasks #35749

Uh oh!

mkoeppe commented Jun 9, 2023 •

edited

Loading

Uh oh!

mkoeppe commented Jun 16, 2023

Uh oh!

kwankyu commented Jul 12, 2023

Uh oh!

kwankyu Jul 12, 2023

Uh oh!

mkoeppe Jul 12, 2023

Uh oh!

kwankyu Jul 12, 2023

Uh oh!

kwankyu left a comment

Uh oh!

kwankyu commented Jul 12, 2023

Uh oh!

mkoeppe commented Jul 14, 2023

Uh oh!

vbraun commented Jul 19, 2023

Uh oh!

kwankyu commented Jul 19, 2023

Uh oh!

kwankyu commented Jul 19, 2023

Uh oh!

kwankyu commented Jul 19, 2023

Uh oh!

kwankyu commented Jul 19, 2023

Uh oh!

github-actions bot commented Jul 19, 2023

Uh oh!

Uh oh!

Uh oh!

Add style guide / reference for # optional - sage.... doctest tags, extend sage -t and sage -fixdoctests for modularization tasks #35749

Add style guide / reference for # optional - sage.... doctest tags, extend sage -t and sage -fixdoctests for modularization tasks #35749

Uh oh!

Conversation

mkoeppe commented Jun 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Description

📝 Checklist

⌛ Dependencies

Uh oh!

mkoeppe commented Jun 16, 2023

Uh oh!

kwankyu commented Jul 12, 2023

Uh oh!

kwankyu Jul 12, 2023

Choose a reason for hiding this comment

Uh oh!

mkoeppe Jul 12, 2023

Choose a reason for hiding this comment

Uh oh!

kwankyu Jul 12, 2023

Choose a reason for hiding this comment

Uh oh!

kwankyu left a comment

Choose a reason for hiding this comment

Uh oh!

kwankyu commented Jul 12, 2023

Uh oh!

mkoeppe commented Jul 14, 2023

Uh oh!

vbraun commented Jul 19, 2023

Uh oh!

kwankyu commented Jul 19, 2023

Uh oh!

kwankyu commented Jul 19, 2023

Uh oh!

kwankyu commented Jul 19, 2023

Uh oh!

kwankyu commented Jul 19, 2023

Uh oh!

github-actions bot commented Jul 19, 2023

Uh oh!

Uh oh!

Add style guide / reference for `# optional - sage....` doctest tags, extend `sage -t` and `sage -fixdoctests` for modularization tasks #35749

Add style guide / reference for `# optional - sage....` doctest tags, extend `sage -t` and `sage -fixdoctests` for modularization tasks #35749

mkoeppe commented Jun 9, 2023 •

edited

Loading