The goal of this ticket is to clean up highlighting in the documentation. There are several issues. Right now, we are patching pygments and Sphinx so that they know about the "sage:" prompt. Also, we are telling Sphinx to highlight all code blocks as if they were Python console blocks. There are various other types of code blocks in the Sage docs: Python, Cython, reST, LaTeX, etc., and right now many are not highlighted at all: Sphinx/pygments tries to parse them as the Python console and fails, so does nothing. Even some of the Sage console blocks aren't highlighted correctly, because pygments refuses to parse input like sage: a?: valid IPython but not valid Python.

To improve the situation: Sage's console is based more on IPython than on Python, and IPython comes with a lexer to use when building its own documentation. Furthermore, that lexer explicitly allows for customization of the input prompt. So we can get rid of the prompt patches and use IPython's lexer, and it succeeds at highlighting blocks containing sage: a? and other IPython-type text. The only disadvantage to this is that IPython's lexer doesn't convert <BLANKLINE> in doctest output to a blank line, so we add a Sphinx transform to handle this.

We also mark some non-Python blocks as what they are, to enable their highlighting.

CC: @timokau

Component: packages: standard

Author: John Palmieri, Jeroen Demeyer

Branch/Commit: 83436ea

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/27528