`sage.features.topcom` #37858

mkoeppe · 2024-04-24T05:27:33Z

Part of Add missing Features for optional and experimental SPKGs #35856

We also change how the SPKG index in the reference manual is generated.
index.rst is now a static file that uses .. include directives to read in generated files index_standard.rst, index_optional.rst, etc.

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation and checked the documentation preview.

⌛ Dependencies

github-actions · 2024-04-25T06:37:02Z

Documentation preview for this PR (built with commit 1c6e248; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

kwankyu · 2024-04-26T23:38:22Z

This needs to be updated:

sage/src/doc/bootstrap

Line 163 in 15fcf52

Features

(btw, I think this list should be automatically generated and split from spkgs list and moved to a new location.)

mkoeppe · 2024-04-26T23:46:03Z

Shall we use a sphinx .. include directive for that?

kwankyu · 2024-04-27T00:04:07Z

I don't know. Anyway, this is for a new PR. I may work on it later.

mkoeppe · 2024-04-27T00:19:39Z

Here's an attempt. The list of features is not autogenerated (we don't do that for any Python modules); but at least it's now in an ordinary editable index file.

kwankyu

Thanks.

LGTM.

mkoeppe · 2024-04-27T04:58:28Z

Thank you!

vbraun · 2024-04-28T10:03:59Z

I'm getting

[sagemath_doc_html-none] [spkg-install] sage --docbuild --no-pdf-links reference/modules inventory --no-prune-empty-dirs
[sagemath_doc_html-none] [spkg-install] [spkg     ] The inventory file is in ../../local/share/doc/sage/inventory/en/reference/spkg.
[sagemath_doc_html-none] [spkg-install] Error building the documentation.
[sagemath_doc_html-none] [spkg-install] Traceback (most recent call last):
[sagemath_doc_html-none] [spkg-install]   File "<frozen runpy>", line 198, in _run_module_as_main
[sagemath_doc_html-none] [spkg-install]   File "<frozen runpy>", line 88, in _run_code
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/__main__.py", line 514, in <module>
[sagemath_doc_html-none] [spkg-install]     sys.exit(main())
[sagemath_doc_html-none] [spkg-install]              ^^^^^^
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/__main__.py", line 510, in main
[sagemath_doc_html-none] [spkg-install]     builder()
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/builders.py", line 823, in _wrapper
[sagemath_doc_html-none] [spkg-install]     getattr(DocBuilder, build_type)(self, *args, **kwds)
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/builders.py", line 163, in f
[sagemath_doc_html-none] [spkg-install]     runsphinx()
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/sphinxbuild.py", line 319, in runsphinx
[sagemath_doc_html-none] [spkg-install]     sys.stderr.raise_errors()
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/sphinxbuild.py", line 255, in raise_errors
[sagemath_doc_html-none] [spkg-install]     raise OSError(self._error)
[sagemath_doc_html-none] [spkg-install] OSError: /home/release/Sage/src/doc/en/reference/spkg/index.rst:28: WARNING: toctree contains reference to nonexisting document 'sage/features'

mkoeppe · 2024-04-28T19:34:14Z

@vbraun Is that in an incremental build? This looks like #37786.

Anyway I have pushed a possible fix

vbraun · 2024-04-29T07:57:18Z

Yes, incremental

vbraun · 2024-05-01T08:17:44Z

Still fails. Try

./bootstrap
cd src/doc
make doc-inventory--reference-spkg     # fail
make doc-inventory--reference-spkg     # succeeds

mkoeppe · 2024-06-09T03:11:57Z

I can't reproduce a failure

vbraun · 2024-06-09T20:49:11Z

I'm getting

[sagemath_doc_html-none] [spkg-install] Error building the documentation.
[sagemath_doc_html-none] [spkg-install] Traceback (most recent call last):
[sagemath_doc_html-none] [spkg-install]   File "<frozen runpy>", line 198, in _run_module_as_main
[sagemath_doc_html-none] [spkg-install]   File "<frozen runpy>", line 88, in _run_code
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/__main__.py", line 530, in <module>
[sagemath_doc_html-none] [spkg-install]     sys.exit(main())
[sagemath_doc_html-none] [spkg-install]              ^^^^^^
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/__main__.py", line 526, in main
[sagemath_doc_html-none] [spkg-install]     builder()
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/builders.py", line 823, in _wrapper
[sagemath_doc_html-none] [spkg-install]     getattr(DocBuilder, build_type)(self, *args, **kwds)
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/builders.py", line 163, in f
[sagemath_doc_html-none] [spkg-install]     runsphinx()
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/sphinxbuild.py", line 319, in runsphinx
[sagemath_doc_html-none] [spkg-install]     sys.stderr.raise_errors()
[sagemath_doc_html-none] [spkg-install]   File "/home/release/Sage/src/sage_docbuild/sphinxbuild.py", line 255, in raise_errors
[sagemath_doc_html-none] [spkg-install]     raise OSError(self._error)
[sagemath_doc_html-none] [spkg-install] OSError: /home/release/Sage/src/doc/en/reference/spkg/index.rst:28: WARNING: toctree contains reference to nonexisting document 'sage/features'

vbraun · 2024-06-09T22:39:15Z

Looks like this is a parallel build race, works when building in a single thread. But make doc-clean && make -j10 doc-html errors out reliably for me. One of the parallel tasks prints a line

[sagemath_doc_html-none] [spkg-install] Deleting empty directory /home/release/Sage/src/doc/en/reference/spkg/sage/features

Seems like the doc-inventory--reference-spkg target conflicts

mkoeppe · 2024-06-09T22:40:31Z

Thanks for the details. I'll try to repro

kwankyu · 2024-06-10T05:43:27Z

@vbraun What is your system?

kwankyu · 2024-06-10T07:35:23Z

@vbraun Would you try #38185? This time with make doc-clean doc-uninstall.

sagemathgh-38185: Simplify doc build process by removing empty directories locally       and thus hopefully fixes the mysterious issue sagemath#37858 (comment). ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies sagemath#37858    URL: sagemath#38185 Reported by: Kwankyu Lee Reviewer(s):

- Part of sagemath#35856 We also change how the SPKG index in the reference manual is generated. `index.rst` is now a static file that uses `.. include` directives to read in generated files `index_standard.rst`, `index_optional.rst`, etc. ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#37858 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

kwankyu · 2024-06-13T04:41:16Z

@vbraun Does #38185 work well with you? Then it needs review.

vbraun · 2024-06-14T22:58:44Z

Worked for me

kwankyu · 2024-06-14T23:28:35Z

Great! Thanks.

vbraun · 2024-06-15T23:06:24Z

Actually no, still happens upon further testing

kwankyu · 2024-06-15T23:17:52Z

It is hard to "fix" an invisible issue. So what is your system and reproducing procedure?

sagemathgh-38185: Simplify doc build process by removing empty directories locally       and thus hopefully fixes the mysterious issue sagemath#37858 (comment). ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies sagemath#37858    URL: sagemath#38185 Reported by: Kwankyu Lee Reviewer(s):

vbraun · 2024-06-16T14:32:28Z

The following steps reproducably cause the failure for me:

$ git clean -f -d -x ./src/doc/en/reference/spkg && ./bootstrap && touch src/doc/en/reference/spkg/index.rst  && ./sage --docbuild --no-pdf-links reference/spkg inventory --no-prune-empty-dirs
[...]
OSError: /home/release/Sage/src/doc/en/reference/spkg/index.rst:28: WARNING: toctree contains reference to nonexisting document 'sage/features'

Running the docbuild command a second time succeeds, then

$ ./sage --docbuild --no-pdf-links reference/spkg inventory --no-prune-empty-dirs
[spkg     ] updating environment: 31 added, 1 changed, 0 removed
[spkg     ] The inventory file is in local/share/doc/sage/inventory/en/reference/spkg.

kwankyu · 2024-06-16T22:27:19Z

Don't you get the same failure if you replace spkg with coding everywhere? I do.

There may be some bug hidden in the doc build procedure in general...

Strangely, this

$ git clean -f -d -x ./src/doc/en/reference/coding && touch src/doc/en/reference/coding/index.rst && ./sage --docbuild --no-pdf-links --no-prune-empty-dirs reference/coding inventory

fails every other time...

kwankyu · 2024-06-16T23:55:20Z

After more experiments, I now realized there is nothing strange in the above experiments. We are breaking assumptions by git clean -f -d -x ./src/doc/en/reference/spkg (or coding). No fault on the doc build mechanism.

The doc build starts initially with the state made by make doc-clean (cleaning src/doc/sage) and make doc-uninstall (removing local/share/doc). This should work

$ make doc-uninstall && git clean -f -d -x ./src/doc/en/reference/spkg && ./bootstrap && touch src/doc/en/reference/spkg/index.rst  && ./sage --docbuild --no-pdf-links reference/spkg inventory --no-prune-empty-dirs

reliably.

Not a bug unless you see it after make doc-clean doc-uninstall

vbraun · 2024-06-17T17:23:38Z

I've seen it after make doc-clean doc-uninstall (because that is in my merge / incremental build script), I just don't have a repro. But it is out there. I run into this issue occasionally when making incremental builds.

And the issue is that dependencies of docbuild artifacts are apparently insufficient.

I think this branch exacerbates the issue because it places /doc/en/reference/spkg/index.rst under version control. So if you frequently change branches its easy to update timestaps which triggers incremental rebuilds, and then if you are unlucky docbuild trips over missing dependencies.

kwankyu · 2024-06-17T20:37:31Z

I just now met the bug. As you say, It seems related with switching branches.

kwankyu · 2024-06-18T13:39:43Z

Makefile

@@ -173,7 +173,7 @@ distclean: build-clean
 bootstrap-clean:
 	rm -rf config/install-sh config/compile config/config.guess config/config.sub config/missing configure build/make/Makefile-auto.in
 	rm -f src/doc/en/installation/*.txt
-	rm -rf src/doc/en/reference/spkg/*.rst
+	find src/doc/en/reference/spkg -name index.rst -prune -o -name *.rst -exec rm -f {} \+
 	for a in environment environment-optional src/environment src/environment-dev src/environment-optional; do rm -f $$a.yml $$a-3.[89].yml $$a-3.1[0-9].yml; done


This line is the source of the "bug" Volker was experiencing.

Unlike previous rm -rf src/doc/en/reference/spkg/*.rst, where the option -r is really ineffective, the find command searches the directory src/doc/en/reference/spkg and its subdirectories, and removes all *.rst files. In particular the file src/doc/en/reference/spkg/sage/features.rst is removed.

The file is in control of the doc builder and to be updated by it if necessary. Suppose that the file is created by the doc builder in other git branch. After switching to this PR branch, ./bootstrap removes the file, but the doc builder still believes the file is intact, and the removed file is not recreated. Hence the boom.

Incremental doc build mixed with switching git branches is inherently instable. But the changed behavior caused by this PR (the command in line 176) exacerbates the instability, as Volker could see through.

Thanks a lot for this detective work!

Thanks. Works well with me (that is, no more the bug for me).

…ndex.rst

…. include'

…igure tarball

…rectories of src/doc/en/reference/spkg

mkoeppe · 2024-06-19T16:39:22Z

Thanks!

kwankyu approved these changes Apr 27, 2024

View reviewed changes

kwankyu added s: positive review and removed s: needs review labels Apr 27, 2024

vbraun added the s: needs work label Apr 28, 2024

vbraun added the s: needs work label May 1, 2024

vbraun force-pushed the develop branch from eba5e19 to e5f42fa Compare June 3, 2024 22:15

github-actions bot added the v: moderate label Jun 9, 2024

vbraun added the s: needs work label Jun 9, 2024

kwankyu mentioned this pull request Jun 10, 2024

Simplify doc build process by removing empty directories locally #38185

Merged

5 tasks

vbraun removed the s: needs work label Jun 14, 2024

vbraun added the s: needs work label Jun 15, 2024

kwankyu reviewed Jun 18, 2024

View reviewed changes

Matthias Koeppe added 6 commits June 18, 2024 09:28

sage.features.topcom: New

7212185

PointConfiguration._have_TOPCOM: Handle FeatureNotPresentError

cd2eb15

src/doc/bootstrap: Only generate parts of src/doc/en/reference/spkg/i…

aedbf61

…ndex.rst

src/doc/en/reference/spkg/index.rst: No longer auto-generated, use '.…

21cc69a

…. include'

bootstrap: Do not put src/doc/en/reference/spkg/index.rst in the conf…

b2298a6

…igure tarball

bootstrap, Makefile (bootstrap-clean): Do not recurse into autodoc di…

1c6e248

…rectories of src/doc/en/reference/spkg

kwankyu added s: positive review and removed s: needs review labels Jun 18, 2024

vbraun merged commit b7ae5f9 into sagemath:develop Jul 24, 2024

github-actions bot removed the s: positive review label Jul 24, 2024

Uh oh!

sage.features.topcom #37858

sage.features.topcom #37858

Uh oh!

Conversation

mkoeppe commented Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Checklist

⌛ Dependencies

Uh oh!

github-actions bot commented Apr 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwankyu commented Apr 26, 2024

Uh oh!

mkoeppe commented Apr 26, 2024

Uh oh!

kwankyu commented Apr 27, 2024

Uh oh!

mkoeppe commented Apr 27, 2024

Uh oh!

kwankyu left a comment

Choose a reason for hiding this comment

Uh oh!

mkoeppe commented Apr 27, 2024

Uh oh!

vbraun commented Apr 28, 2024

Uh oh!

mkoeppe commented Apr 28, 2024

Uh oh!

vbraun commented Apr 29, 2024

Uh oh!

vbraun commented May 1, 2024

Uh oh!

mkoeppe commented Jun 9, 2024

Uh oh!

vbraun commented Jun 9, 2024

Uh oh!

vbraun commented Jun 9, 2024

Uh oh!

mkoeppe commented Jun 9, 2024

Uh oh!

kwankyu commented Jun 10, 2024

Uh oh!

kwankyu commented Jun 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwankyu commented Jun 13, 2024

Uh oh!

vbraun commented Jun 14, 2024

Uh oh!

kwankyu commented Jun 14, 2024

Uh oh!

vbraun commented Jun 15, 2024

Uh oh!

kwankyu commented Jun 15, 2024

Uh oh!

vbraun commented Jun 16, 2024

Uh oh!

kwankyu commented Jun 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwankyu commented Jun 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vbraun commented Jun 17, 2024

Uh oh!

kwankyu commented Jun 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwankyu Jun 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkoeppe Jun 18, 2024

Choose a reason for hiding this comment

Uh oh!

kwankyu Jun 18, 2024

Choose a reason for hiding this comment

Uh oh!

`sage.features.topcom` #37858

`sage.features.topcom` #37858

mkoeppe commented Apr 24, 2024 •

edited

Loading

github-actions bot commented Apr 25, 2024 •

edited

Loading

kwankyu commented Jun 10, 2024 •

edited

Loading

kwankyu commented Jun 16, 2024 •

edited

Loading

kwankyu commented Jun 16, 2024 •

edited

Loading

kwankyu commented Jun 17, 2024 •

edited

Loading

kwankyu Jun 18, 2024 •

edited

Loading