In the current state it is not going to pass any tests due to not handled values in switch().

Some minimal language is finished: supporting unary/binary plus and calling/nesting parenthesis alongside with a single actual value (register) makes me think I can proceed with extending current parser code to accept the rest of the language and not inventing something else. On the way is, in any order

1. checking whether I got handling operators precedence right by adding multiplication (others may wait until 3. is sorted out)
2. comma operator (it is simpler to express multiargument call as Call(func, Comma(Comma(arg1, arg2), arg3)), then use comma code for list and dictionary literals with minimal modifications)
3. { handling as the greatest ambiguity in VimL expressions: it is used for lambdas, dictionaries and for curly braces names at once, in some positions exactly any one of the listed entities may appear
4. actually getting completion working: my idea is that if completion is requested parser function needs to be supplied with what is there up to the cursor, at EOC parser will set up completion by analyzing what is there in the AST stack and what was the previous token
5. colon operator (either Ternary(cond, Colon()) or for dictionary, same reasoning as for commas).

Rest of parsing should be handled only after implementing the above.

(BTW, I think it may be a good and relatively easy to test lexer with KLEE. The only thing I completely do not like is that they do not support llvm-4.0 yet and llvm-3.x is not slotted and thus may not be installed alongside with 4.0, so KLEE could only either run in docker on my machine or I need to compile LLVM myself.)

Managed to get KLEE run some tests, though it did not find anything useful yet. At least, I could be made sure that lexer always skips something (i.e. will not make parser fall into the infinite loop) and does not crash. Should be more useful for the parser (when it is finished), also turning code which compiles for KLEE into an executable suitable for fuzzing is much easier.

The KLEE finally finished, no errors in lexer found. For some reason I did not have time automatically reported (zsh has a feature which makes it automatically act as if command was run like time … if it took more then $REPORTTIME seconds to finish which I have set to 5, with some other exceptions which should not apply to my script), but first file generated by the script (build/klee/a.bc) states to be created Sep 12 01:21, the last one (build/klee/out/info) states Sep 12 16:04: almost 15h.

Total generated 53 873 “test cases”, 221 MiB according to du -hs. Generated directory is greatly compressable: attached tar.xz archive is only 1.3 MiB. Archive attached in case somebody is interested what KLEE generates, though it is not very interesting: lots of “test cases”, LLVM assembly used (plus bitcode), three things looking like log files, some coverage data in text format (not very readable) and KLEE own profiling data (rather top level, KLEE was being used not for profiling KLEE after all).

klee.tar.xz.not.zip

@ZyX-I so you plan to expose the AST somehow, e.g. as a dictionary , to plugins? Or is it possible?

I plan to expose it, but only AST (not lexer). In the future - via two functions, one for parsing expressions exclusively, like eval() (it has some differences regarding newline handling), another for parsing Ex commands.

And only via API, relying on the VimL function generator for VimL. Plus parser may be exposed via ASYNC function.

Got working figure braces (dictionary, curly-braces-names in value position, lambdas, but not complex curly-braces-names setups like foo{"bar"}) and commas (minus multiargument calls: should be working but not tested yet); operator priority code looks like something working as well (minus “no associativity” part). Most problematic parts to code in parser should be coded now, though there are still challenges like highlighting of string literals, floating-point numbers, ternary operator or non-associative operators (and, of course, making that pass linter which will not be happy with a big function out there).

And no completion code yet, though I have an idea how to code it which needs only minor modifications to what is already there.

Error while building with ninja:

CMake Error at /home/oni-link/git/neovim/cmake/RunXgettext.cmake:13 (message):
  xgettext failed to run correctly: /usr/bin/xgettext: Non-ASCII string at
  ../viml/parser/expressions.c:1274.

                     Please specify the source encoding through --from-code.

Message contains ’:

                ERROR_FROM_NODE_AND_MSG(
                    new_top_node,
                    _("E15: Don’t know what figure brace means: %.*s"));

Got working lambdas, dictionaries, curly braces names, function calls and nesting parenthesis, ternary operator, comparisons, unary/binary plus, operator associativity in general.

What is left of specifically the parser

• Dot handling.
• String literals handling.
• Finishing rest of “not interesting” operators.
• Determining completion context.
• Making all that pass the linter (it will not be happy to see that huge parser function).
• KLEE check (going to be very time-consuming; currently rerunning it for the lexer).

Well, it appears that &isident … is responsible for determining identifier length inside <…>. Since I set it to an empty string for testing only - could be present inside <…> then, and only because hyphen-minus is being tested for separately. I think that this is a bug because it is not mentioned in documentation and I see no good reasons for this option to affect string parsing.

Got that after building libnvim-test on travis:

+asan_check /home/travis/build/neovim/neovim/build/log
+check_logs /home/travis/build/neovim/neovim/build/log '*san.*'
++find /home/travis/build/neovim/neovim/build/log -type f -name '*san.*'
+for log in '$(find "${1}" -type f -name "${2}")'
+sed -i /home/travis/build/neovim/neovim/build/log/ubsan.28775 -e '/Warning: noted but unhandled ioctl/d' -e '/could cause spurious value errors to appear/d' -e '/See README_MISSING_SYSCALL_OR_IOCTL for guidance/d'
+local err=
++find /home/travis/build/neovim/neovim/build/log -type f -name '*san.*' -size +0
+for log in '$(find "${1}" -type f -name "${2}" -size +0)'
+cat /home/travis/build/neovim/neovim/build/log/ubsan.28775
==28775==LeakSanitizer has encountered a fatal error.
==28775==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==28775==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)

What is that?

In any case, I am restarting travis under assumption that problem is temporary. Other CIs succeeded.

With verboser logging:

+cat /home/travis/build/neovim/neovim/build/log/ubsan.28765
==28765==AddressSanitizer: failed to intercept '__isoc99_printf'
==28765==AddressSanitizer: failed to intercept '__isoc99_sprintf'
==28765==AddressSanitizer: failed to intercept '__isoc99_snprintf'
==28765==AddressSanitizer: failed to intercept '__isoc99_fprintf'
==28765==AddressSanitizer: failed to intercept '__isoc99_vprintf'
==28765==AddressSanitizer: failed to intercept '__isoc99_vsprintf'
==28765==AddressSanitizer: failed to intercept '__isoc99_vsnprintf'
==28765==AddressSanitizer: failed to intercept '__isoc99_vfprintf'
==28765==AddressSanitizer: failed to intercept '__cxa_throw'
==28765==AddressSanitizer: libc interceptors initialized
|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap  ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem     ||
MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
thread_local_quarantine_size_kb=1024K
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 0x7fff8000
==28765==Installed the sigaction for signal 11
==28765==Installed the sigaction for signal 7
==28765==Installed the sigaction for signal 8
==28765==T0: stack [0x7ffdf1066000,0x7ffdf1866000) size 0x800000; local=0x7ffdf1864260
==28765==LeakSanitizer: Dynamic linker not found. TLS will not be handled correctly.
==28765==AddressSanitizer Init done
==28766==Could not attach to thread 28765 (errno 1).
==28766==Failed suspending threads.
==28765==LeakSanitizer has encountered a fatal error.
==28765==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==28765==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)

The problem somehow solved itself.

New build failure looks unrelated:

./test/functional/helpers.lua:268: 
retry() attempts: 2
./test/functional/ui/screen.lua:302: Row 2 did not match.
Expected:
  |{1: }                                                 |
  |*lost                                              |
  |gained                                            |
  |{4:~                                                 }|
  |{5:[No Name] [+]                                     }|
  |:                                                 |
  |{3:-- TERMINAL --}                                    |
Actual:
  |{1: }                                                 |
  |*gained                                            |
  |lost                                              |
  |gained                                            |
  |{5:[No Name] [+]                                     }|
  |:                                                 |
  |{3:-- TERMINAL --}                                    |

To print the expect() call that would assert the current screen state, use
screen:snapshot_util(). In case of non-deterministic failures, use
screen:redraw_debug() to show all intermediate screen states.  

stack traceback:
	./test/functional/helpers.lua:268: in function 'retry'
	...uild/neovim/neovim/test/functional/terminal/tui_spec.lua:300: in function <...uild/neovim/neovim/test/functional/terminal/tui_spec.lua:293>

CI succeeded.

FYI the #7234 (comment) failure is my fault, I will make another change to the TUI startup and see if it continues; if it continues I will change the test to avoid spurious failure.

@ZyX-I what is your plan for command parsing or general VimL parsing? Will this be a new API function that accepts entire sources?

Is there a need for nvim_parse_* or would it make sense to have just one API function: nvim_parse .

@justinmk There are use-cases for separate expression parser in the API other then making it easier to test some things (I mean, test as a plugin developer, Neovim developers are always going to have unit tests available), you may see a list in the documentation of the new function. For VimL parser I plan on exactly the API function accepting entire source, though internally it should be possible to save state and use it later to proceed (needed for alternative syntax highlighting: if we have a parser and highlighter in a C code, why limit highligter to command-line only?).