Skip to content

Conversation

teto
Copy link
Member

@teto teto commented Mar 15, 2017

In order to generate a "palette" (https://github.com/teto/nvim-palette ; related issue #6240), I moved the one-liner option descriptions from quickref.txt to options.lua so that it can be reused more easily.
In order not to have duplicated strings across the source, I generate the incriminated quickref help from options.lua.

As a summary of changes:

  • options in options.lua now have a short_desc= member.
  • genoptions.lua has 3 distinct targets: doc, csv, header
    • header generates the C header (its original role)
    • csv converts to CSV the lua table. This is what I used for nvim-palette. As such it's not critical to nvim so it could be either removed or moved to contrib/
    • doc generates quickref.txt from quickref*.tpl (tpl=template) files. It generates quickref_10.txt and does cat quickref_*.tpl > quickref.txt. That was the only thing I could think with my skillset (apart from using jinja templates). This leaves the tpl files in the runtime folder but I wanted to know if you had another idea in mind before cleaning that up. There could be value in generating other parts of the doc so if we could standardize the process that might be nice.
  • I modified the xgettext cmake launcher so that it fetches these descriptions as well. vimscript (Feature: Gettext support for Vimscript. #1453) and lua gettext support look bad but for remote plugins such as python, that could be a good way to retrieve the strings.

@justinmk
Copy link
Member

justinmk commented Mar 15, 2017

Nice.

  • Can't we avoid templates? Why can't it just look for the text in quickref.txt between a start tag and end tag, then delete that text and replace it? That is what scripts/gen_api_vimdoc.py does for api.txt (search for delete_lines_below).
  • Instead of CSV: JSON or msgpack?
  • Haven't looked at the code. How do plugins get this data from nvim? Or what's the plan?

if name == "scope" then
-- print("SCOPE")
append = toCSV(opt[name])
else
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2-space indent. In general, these scripts should follow the typical guidelines.

@ZyX-I
Copy link
Contributor

ZyX-I commented Mar 15, 2017

Maybe it is time to add some templating engine as a dependency? Various generators could be far more readable with good engine. I do not think that avoiding templates is a good idea: when I was checking whether it would be easy to add varargs I wished to rewrite gendispatch.lua with something: proper template engine allows to

  1. Naturally maintain indentation. How this is done now is crap which is easy to accidentally make wrong. Not that compiler cares, but I remember myself writing something to gendispatch.lua (lua bindings AFAIR), checking the result and going to do the corrections.
  2. Makes it rather obvious what the result would look like without actually looking at the result.
  3. Is more compact.

Also it would not require making generated dispatch file be *.h include-only “header” file and not proper *.c file able to pass IWYU check in the future.

@justinmk
Copy link
Member

In this case at least (and I suspect many future ones seeking to use template rather than "obvious"), template makes things worse. Now we have a missing runtime/doc/quickref.txt file in the tree, and people have to start searching elsewhere for non-generated help docs. runtime/doc/quickref.txt should stay in the obvious place. The generator should only replace the text that's needed.

@@ -0,0 +1,598 @@
*quickref.txt* For neovim version 0.2. Last change: 2017 Mar 13
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs always mention "Nvim" or "nvim" only. Should stay consistent.

@ZyX-I
Copy link
Contributor

ZyX-I commented Mar 15, 2017

Just because *.txt file is *.txt it does not mean it can’t be a template. And template code is basically a thing which will be replaced by the generator.

@teto
Copy link
Member Author

teto commented Mar 15, 2017

Instead of CSV: JSON or msgpack?
Haven't looked at the code. How do plugins get this data from nvim? Or what's the plan?

Right now it's offline, plugin need to embed their own listing of descriptions. I chose csv but it could be anything. I wish users could query nvim for setting descriptions but I couldn't find how to justify nvim advertising option descriptions except for my use case so I left that out. I am open to suggestions if you think advertising (via msgpack) the short descriptions is ok.

I agree with @ZyX-I for templating. I am not happy with my solution but I see other parts of the doc that could be generated and I see value in using some templating (with rules that make it understandable).

@justinmk
Copy link
Member

justinmk commented Mar 16, 2017

It's similar use-case as #6123, right? The data generated from genoptions.lua can be shipped as a messagepack runtime file, e.g.:

runtime/data/options-en.mpack
runtime/data/options-fr.mpack
...

This will be a messagepack blob, any plugin can load it and inspect it via msgpackparse(). We could do the same thing for #6123 (runtime/data/mappings-{locale}.mpack). cc @tjdevries

To generate the messagepack data, genoptions.lua can emit JSON and feed it to nvim's json_decode() => msgpackdump().

@tjdevries
Copy link
Contributor

Yes, this is some of the exact type of things I was looking to do. Once we have the descriptions for some of the mappings, we could use the same process to generate mpack items for that.

I definitely think JSON is a much stronger candidate than CSV. Also, it makes sense to use JSON as justinmk says so that we could dump it into mspack. Once you have these options available, people will find ways to use them introspectively.

@teto
Copy link
Member Author

teto commented Mar 18, 2017

I've added dkjson as a dependancy (it exists on luarocks) to export the lua table to json. Then I generate an options.mpack but do you want one per language ? translations are available in the mo files already so that may be redundant but on the other hand, one mpack per lang is more straightforward (I have no opinion there). Also in the first case plugins would need to know the gettext path, aka #define LOCALE_INSTALL_DIR "/usr/local/share/locale" which does not seem exported.
I am very tempted to adopt jinja2 http://jinja.pocoo.org/ as a template engine since I can see other places in the doc we could generate from gettext (auevents.lua comes to mind, help header with version number).

@justinmk
Copy link
Member

Jinja requires python. Adding python as a build dependency for non-CI tasks is too much. We already have too many dependencies.

@ZyX-I
Copy link
Contributor

ZyX-I commented Mar 18, 2017

Why genoptions.lua should emit json and feed it somewhere? We already have msgpack generator dependency.

About jinja: this adds Python as a required build time dependency and requires to rewrite some scripts in Python. Better find something based on lua, though I did not find a generic (non-HTML) templating engine which is not stagnating.

And note that templates should be used not only for documentation: there are files like dispatch_wrappers.generated.h or syntax/vim/generated.vim, especially the second (note that it requires wrapping with syn keyword repeated on each line). Better if template engine is extensible with something custom, it does not have to have all features we may need out of the box, cycles, if and substituting results of expression evaluation are enough for the start.

I would actually be happier if help files alongside with docstring comments were replaced with sphinx documentation (it is even possible to create a sphinx target which will generate Vim help files out of rst files): unlike doxygen html documentation from sphinx does not look like crap and is more configurable and sphinx itself is easier to extend: up to writing the functions which will generate documentation like we want now with templates or creating vimhelp target. But that would be too much work and not too much benefit. Linux kernel started moving towards documenting everything with sphinx though.

@ZyX-I
Copy link
Contributor

ZyX-I commented Mar 18, 2017

Python is already there on most *nix systems. It would be OK if we did not already have everything in lua.

BTW, if we were using sphinx I could suggest to keep options.lua, etc in the documentation itself. But vim help files do not allow even comments, not to mention custom markup constructs.

@teto
Copy link
Member Author

teto commented Mar 18, 2017

Forget about the template engine, I will generate the help "by hand" or leave it as is (does it matter that much anyway ?!).

Why genoptions.lua should emit json and feed it somewhere? We already have msgpack generator dependency.

What do you mean ? we pull a lua-msgpack already (could not find 'lua-MessagePack' ) ? or use the nvim binary to generate the mpack ?

Btw what about the question of generating one mpack per language vs just generate the US mpack and let plugins deal with gettext ?

@ZyX-I
Copy link
Contributor

ZyX-I commented Mar 18, 2017

@teto We pull in mpack lua package, see how gendispatch.lua uses it. Nvim binary theoretically can be used for both JSON and msgpack, but on your place I would not be much fond of this idea: this means that you need to somehow transfer data to nvim which without #4411 will include serializing data in lua. Probably implicitly serializing: if you trasfer data to nvim like functional tests do you will inevitably serialize data with the very mpack lua library I am suggesting.

@ZyX-I
Copy link
Contributor

ZyX-I commented Mar 18, 2017

About mpack per language/US mpack: on first iteration I would suggest to just use US mpack. If somebody needs localized descriptions he may ask us for it. Maybe by the time there will be access to gettext through VimL: currently you could not easily use gettext to localize your plugins.

@teto
Copy link
Member Author

teto commented Mar 18, 2017

To sum up the changes:

  • I discarded the doc generation part. That can be done in another PR. I am not excited by recreating a template engine.
  • genoptions.lua accepts 2 commands as argv[1]: "header" and "mpack", mpack generates runtime/data/options.mpack (using libmpack)
  • options.lua is parsed by xgettext
  • while trying to generate localized options-{lang}.mpack, I had to let the .mo files adopt a gettext structure in build/locale/lang/LC_MESSAGES/nvim.mo. I didn't revert that as it can be useful in the future and makes more sense imo then the current way of generating those in $PROJECT_SOURCE/src/nvim/po

When trying to update pos ninja -C build update-po, I get "Escape characters not allowed":

ninja: Entering directory `build'
[2/28] Updating cs.cp1250.po
FAILED: src/nvim/po/CMakeFiles/update-po-cs.cp1250 
cd /home/teto/neovim2/build/src/nvim/po && /usr/bin/cmake -DICONV_PRG=/usr/bin/iconv -DINPUT_FILE=/home/teto/neovim2/src/nvim/po/cs.po -DOUTPUT_FILE=/home/teto/neovim2/src/nvim/po/cs.cp1250.po -DINPUT_ENC=ISO-8859-2 -DOUTPUT_ENC=cp1250 -DOUTPUT_CHARSET=cp1250 -P /home/teto/neovim2/cmake/ConvertPo.cmake
CMake Error at /home/teto/neovim2/cmake/ConvertPo.cmake:10 (message):
  iconv failed to run correctly: /usr/bin/iconv: séquence d'échappement non
  permise à la position 32

[4/28] Updating sk.cp1250.po
FAILED: src/nvim/po/CMakeFiles/update-po-sk.cp1250 
cd /home/teto/neovim2/build/src/nvim/po && /usr/bin/cmake -DICONV_PRG=/usr/bin/iconv -DINPUT_FILE=/home/teto/neovim2/src/nvim/po/sk.po -DOUTPUT_FILE=/home/teto/neovim2/src/nvim/po/sk.cp1250.po -DINPUT_ENC=ISO-8859-2 -DOUTPUT_ENC=cp1250 -DOUTPUT_CHARSET=cp1250 -P /home/teto/neovim2/cmake/ConvertPo.cmake
CMake Error at /home/teto/neovim2/cmake/ConvertPo.cmake:10 (message):
  iconv failed to run correctly: /usr/bin/iconv: séquence d'échappement non
  permise à la position 782

@teto
Copy link
Member Author

teto commented Mar 20, 2017

It would be neat if a US native & longstanding vimmer could check & update the short_desc in options.lua.

@teto teto changed the title [WIP] Add short descriptions to options.lua [RFC] Add short descriptions to options.lua Mar 20, 2017
@marvim marvim added RFC and removed WIP labels Mar 21, 2017
@teto teto force-pushed the short_desc branch 2 times, most recently from 6078ad4 to 882451d Compare April 3, 2017 14:24
@teto teto changed the title [RFC] Add short descriptions to options.lua [RDY] Add short descriptions to options.lua Apr 3, 2017
@marvim marvim added RDY and removed RFC labels Apr 3, 2017
@teto
Copy link
Member Author

teto commented Sep 16, 2017

Just as a precision, I reverted it from RDY to WIP because even though the .mpack is generated, it can be impractical to work with as for example the scope is typically dumped as strings ("buffer", "window" ). For instance in https://github.com/teto/nvim-palette, I do some preprocessing on it (and as I don't saved the cached product, the plugin ends up being slow). Might be best postponed to 0.2.2

@teto
Copy link
Member Author

teto commented Mar 1, 2020

I am ready to resume work on this and the associated plugin. With the recent lua advances (type validation etc), I thought we could envision something more comprehensive aka having plugins describe their own settings in an optional spec and vim capable of exporting this.
with an API:
export_options({})
export_plugin_options({})
It could return either a concatenation of files, an mpack file etc.

Example of a spec (json or csv may be simpler or we could just)

{
	{ "name": "g:sneak#prompt",
      "type": 'string',
      "description": "Prompt when triggering vim-sneak"
      "validator": "some lua code"
	},
	{
		...
	}
}

@teto
Copy link
Member Author

teto commented Sep 25, 2020

@tjdevries @sunjon (the telescope team) that's what I used for my palette if you wanna play with it :p

@sunjon
Copy link

sunjon commented Sep 25, 2020

@tjdevries @sunjon (the telescope team) that's what I used for my palette if you wanna play with it :p

ah great, I'd not thought exactly how I was going to scrape this info. This is very helpful.

@teto teto changed the title [WIP] Add short descriptions to options.lua [RFC] Add short descriptions to options.lua Sep 25, 2020
@teto
Copy link
Member Author

teto commented Sep 27, 2020

@sunjon made me realize I did not share my source of truth, which is https://github.com/neovim/neovim/blob/master/runtime/optwin.vim .
Also the enumerated values at https://github.com/neovim/neovim/blob/master/src/nvim/option.c#L290 could also be moved to options.lua

@teto teto mentioned this pull request Oct 31, 2020
@teto
Copy link
Member Author

teto commented Nov 10, 2020

would you be ok with just merging the descriptions without the rest ?
I am sure it will come in handy and will increase discovery, it's already used in
nvim-telescope/telescope.nvim#133 .

@bfredl
Copy link
Member

bfredl commented Nov 10, 2020

As a consequence of merging this can we officially untruth https://github.com/neovim/neovim/blob/master/runtime/optwin.vim ? because no one who does options in neovim has the time (nor should need the time) to care about that file.

add one-liner option descriptions in src/nvim/options.lua
They are taken from optwin.vim and should be easier to use in a
programmatic manner, for instance in a "palette", like in telescope
`:Telescope vim_options`.
@teto teto merged commit 0a95549 into neovim:master Nov 10, 2020
@teto teto deleted the short_desc branch November 10, 2020 23:06
tjdevries pushed a commit to tjdevries/neovim that referenced this pull request Nov 11, 2020
add one-liner option descriptions in src/nvim/options.lua
They are taken from optwin.vim and should be easier to use in a
programmatic manner, for instance in a "palette", like in telescope
`:Telescope vim_options`.
tjdevries pushed a commit to tjdevries/neovim that referenced this pull request Nov 11, 2020
add one-liner option descriptions in src/nvim/options.lua
They are taken from optwin.vim and should be easier to use in a
programmatic manner, for instance in a "palette", like in telescope
`:Telescope vim_options`.
@justinmk justinmk changed the title [RFC] Add short descriptions to options.lua Add short descriptions to options.lua May 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants