PR commits start with “ex_getln: Clean up draw_cmdline a bit”.

Example code which will highlight using pygments:

# color_cmdline.py
def color_cmdline_setup():
    import vim
    from pygments import lex, format
    from pygments.lexers import get_lexer_by_name
    from pygments.formatter import Formatter
    from pygments.token import Token

    try:
        from __builtin__ import unicode
    except ImportError:
        unicode = str

    COLORS = {
        Token:                    'Normal',

        Token.Comment:            'Comment',
        Token.Comment.Preproc:    'PreProc',
        Token.Keyword:            'Keyword',
        Token.Keyword.Type:       'Type',
        Token.Operator:           'Operator',
        Token.Name:               'Identifier',
        Token.Name.Builtin:       'Statement',
        Token.Name.Function:      'Function',
        Token.Name.Constant:      'Number',
        Token.String:             'String',
        Token.Number:             'Number',
        Token.Punctuation:        'Operator',

        Token.Error:              'Error',
    }

    class NvimFormatter(Formatter):
        def __init__(self, **kwargs):
            super(NvimFormatter, self).__init__(**kwargs)
            self.offset = 0
            self.colorscheme = kwargs.get('colorscheme', None) or COLORS

        def format_unencoded(self, tokensource, outfile):
            for ttype, value in tokensource:
                ttype_str = str(ttype).replace(' ', '_')
                value = value.rstrip('\n')
                color = self.colorscheme.get(ttype)
                while color is None:
                    ttype = ttype[:-1]
                    color = self.colorscheme.get(ttype)
                if isinstance(value, unicode):
                    vlen = len(value.encode('utf-8'))
                else:
                    vlen = len(value)
                if vlen:
                    outfile.write('{} {} {} {}\n'.format(
                        self.offset, self.offset + vlen, color, ttype_str))
                self.offset += vlen

    def setup():
        global color_cmdline_highlight
        lexer = get_lexer_by_name('vim')

        def color_cmdline_highlight():
            cmdline = vim.eval('a:cmdline')
            add_tt = False
            if bool(int(vim.eval('get(a:000, 0)'))):
                add_tt = True
            tokens = lex(cmdline, lexer)
            fmt = format(tokens, NvimFormatter(bg=vim.options['background']))
            return [
                ((lambda s, e, g, tt: (
                    (int(s), int(e), g)
                    + ((tt,) if add_tt else ())
                ))(*line.split()))
                for line in fmt.split('\n')[:-1]
            ]

    setup()


color_cmdline_setup()

pyfile <sfile>:h/color_cmdline.py
function PygmentsFormat(cmdline, ...)
  return pyeval('color_cmdline_highlight()')
endfunction
let g:Nvim_color_cmdline = 'PygmentsFormat'

it might be worthwhile to think of how this will interact with #6162 later. I guess GUIs can recieve a list of pairs [text, attrs] with the attrs that apply to that chunk of text, alternatively just the text and the list of[start, stop, attrs].

Actually it appears that input({opts}) is better then requiring prompt. What’s the point in requiring prompt if the official documentation states that you may have no prompt by supplying it an empty string, so basically prompt is not required?

I would guess the purpose is to nudge plugin authors to consider adding a prompt (instead of just calling input() with no args without even checking the docs that a prompt is possible), but with a dict they will need to check the docs anyway so then it makes sense to make it optional.

Also, having user commands and input() executing a callback on every input change is going to be useful for other purposes than coloring the cmdline, for instance live feedback in a window like inccommand. So I wonder if the callback argument really should be called highlight or color, maybe instead callback to acknowledge that it has other potential uses?

@bfredl One of the points of the coloring is “better have no colors then make user problems”. People using that for other purposes should be prepared that Neovim may reject calling their callback if it causes troubles: shows errors, takes too long to finish, alters some state, explicitly disabled by the user, etc. In the PR I am only going to disable it for errors, arabic shaping (not sure how to implement and how to test) and probably for some option set, but that may be extended depending on the problems I spot. If you know an existing facility to automatically interrupt after some time passed I would implement “too long to finish” as well.

alters some state, explicitly disabled by the user, etc. In the PR I am only going to disable it for errors, arabic shaping (not sure how to implement and how to test) and probably for some option set,

This would be fine for other use cases as long as one still can rpcnotify() (not rpcrequest()) and start_timer(0, ...) (currently the canonical way of "do something when the main loop is ready")

If you know an existing facility to automatically interrupt

I don't know if got_int somehow can be abused for this. Checking got_int is quite common but calling os_breakcheck less so (which would be the natural place to check the time limit)

(Without the merge clint fails locally.)

AppVeyor failure looks unrelated:

[  FAILED  ] 1 test, listed below:
[  FAILED  ] C:/projects/neovim/test/functional\core\job_spec.lua @ 69: jobs changes to given `cwd` directory
C:/projects/neovim/test/functional\core\job_spec.lua:78: Expected objects to be the same.
Passed in:
(table) {
  [1] = 'notification'
  [2] = 'stdout'
 *[3] = {
    [1] = 0
   *[2] = {
      [1] = 'C:\projects\neovim\Xtest-tmpdir\nvimdDir0n\0' } } }
Expected:
(table) {
  [1] = 'notification'
  [2] = 'stdout'
 *[3] = {
    [1] = 0
   *[2] = {
      [1] = 'C:\projects\neovim\Xtest-tmpdir\nvimdDir0n\0'
     *[2] = '' } } }
stack traceback:
	C:/projects/neovim/test/functional\core\job_spec.lua:78: in function <C:/projects/neovim/test/functional\core\job_spec.lua:69>

Travis hanged in oldtests, test49:

!echo 'q^Mq' | $NVIM_PRG -u NONE -N -es -c 'debuggreedy|set viminfo+=nviminfo' -c 'let ExtraVimBegin = 1' -c 'let ExtraVimResult = "/tmp/nvimSdwGqV/9"' -c 'breakadd func 3 G' -S /tmp/nvimSdwGqV/8
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

Restarting.

My tentative observation is that the test49 hang seems to persist if travis cache is not cleared.

@justinmk This is planned to be merged after dealing with more urgent issues for the next (0.2.1) release?

@ZyX-I If you think it's ready, I think we should include it in 0.2.1

There's a non-trivial amount of logic (and documentation) involving getln_interrupted_highlight, prompt_id, related to concerns about performance. Just curious, why would performance be noticed here, typically commands occupy only 1 line, and AFAIK vim does not take such measures for insert-mode highlighting.

Insert-mode highlighting is done in C code. I am running VimL and expect it to be slow in cases like somebody trying to implement pygments in pure VimL. And I do not see too non-trivial amount: for getln_interrupted_highlight it is just “if <C-c> happened to arrive while highlighting assume that highlighting appeared to be too slow for the user to be disturbed, so no longer execute the highlight (:try catches interrupts as exceptions, so nothing is done specially here, they go with other errors), but do not assume that user meant to interrupt the prompt (what getln_interrupted_highlight is for)”. prompt_id is there not for performance, it disables running highlight in case of errors (in order to not spam user with errors), but makes the disablement not entirely permanent, only for one prompt.

Or, the other way around: getln_interrupted_highlight is what allows you to interrupt infinite cycle in highlight function, but still run what you want. Without it you could not do let g:Nvim_color_cmdline = function('Halting') and run any Ex mode command afterwards without hacks (there still is <C-r>= and execute()), including the one which will fix the problem.

prompt_id allows disabling highlighting for the duration of one prompt only, in the above case it is needed for you to not type <C-c> after typing each character of unlet g:Nvim_color_cmdline. Though I think I can remove prompt_id and just keep prev_prompt_errors and last_ccline_colors in CmdlineInfo. Behaviour will slightly change: in the example you would need to type second <C-c> if you use another prompt: :u<C-c>nlet <C-r>="g"<CR>:<C-c>Nvim_color_cmdline<CR>, and highlight function will not be recalled.

CI succeeded, only left running is gcov “flaky” build which will not possibly affect green status.

All “flaky” appeared to be green as well, it is uncommon.