lua callbacks for nvim_buf_attach #9170

bfredl · 2018-10-29T14:34:59Z

I want to respond to detailed buffer change events in an in-process lua plugin. Currently these events are available only for rplugins. In general I think we want to accept lua callbacks in API `_attach' methods. Another application that has been discussed is lua code rendering "external" UI elements by subscribing to the high-level UI events.

This patch sketches the following:

local a = vim.api
function thecb(...)
  -- args are "nvim_buf_lines_event", 1, 85, 8, 8, { "" }, false
  print(require'inspect'({..}))
 -- return true to unsubscribe (not implemented yet)
end

a.nvim_buf_attach(0, false, {on_event=thecb})
print(require'inspect'(lastargs))

The usual caveats about incomplete implementation of lowest-effort design apply, suggestions welcome. Some considerations:

This design passes different event kinds to the same callback, an alterative would be to register one callback for update, changedtick and detach events. The later would imply that api.nvim_ui_attach(..., {'on_popupmenu_show': on_show, ...}) implicitly filters out only the interesting events, though we could of course make event subgroups opt-in also for the single callback approach.

The current implementation adds lua references to recursive Object structures, which would affect everything that dispatches over object types, I'm not sure we really want this. An alternatively that doesn't touch Object semantics or go deep into serialization logic, would be to just allow a union{Dictionary,LuaRef} top-level type for parameters to API functions (the visible metadata type will still be Dictionary). Then the function implementation will do all the unpacking of the lua table containing references itself.

Unlike rpc clients, the lua callback can reliably get the changed text using get_lines if it wants to. If the callback doesn't use the actual text, only the range information, the text would have been converted to lua strings for no reason, which would be costly for larger changes. As an example, the tree-sitter library has one API function to mark a region as changed, which one would call in the change event, and another function to actually reparse the tree, which could be done later when the tree is really needed. Though we could keep the same behavior by default, and disable sending text with a flag (which then would be available also for RPC clients).

KillTheMule · 2019-01-24T18:02:17Z

I realize I'm late to the party, but wouldn't it have been a lot easier to add a BufUpdate autocommand and let plugins collect the lines themselves?

bfredl · 2019-01-24T18:30:00Z

@KillTheMule Maybe, it could make sense together with native lua autocmds, which would be a good idea anyway:

hnd = vim.autocmd{name="BufUpdate", buffer=id, cb=function(event)
  -- event contains same data as v:event
end}

Though unnecessary copy of lines can simply be erased without changing anything else, if that is the main concern. Also as mentioned this pattern would be useful also for other things were autocmds cannot be used.. Like support nvim_ui_attach from lua to reimplement TUI popupmenu in lua, which will run code in a context where vimL (thus autocmds) are not safe.

KillTheMule · 2019-01-24T18:41:24Z

No, I wasn't concerned about line copies, and I did not fully understand the scope of this PR :) I just thought about how to get buffer updates to in-process lua, and felt an autocommand would have been easy to implement, that's all.

bfredl · 2019-01-24T19:03:18Z

@KillTheMule easy to implement yes, but using it (without any further addition) would require usage of nvim_command and publicly exported name to register and usage of nvim_eval to get the necessary information in the autocmd impl. A good design goal is to enable lua interface to the core, without any intermediate vimL, no?

justinmk · 2019-04-28T15:10:45Z

src/nvim/buffer.c

@@ -1770,6 +1770,8 @@ buf_T * buflist_new(char_u *ffname, char_u *sfname, linenr_T lnum, int flags)
  buf->b_p_bl = (flags & BLN_LISTED) ? true : false;    // init 'buflisted'
  kv_destroy(buf->update_channels);
  kv_init(buf->update_channels);
+  kv_destroy(buf->update_callbacks);
+  kv_init(buf->update_callbacks);


Rather than adding an extra foo_callbacks field for each "attach" mechanism, would it make sense to have a new internal channel type that forwards channel events to callbacks? Then this could be used to register callbacks for any event type.

I'm not sure it does, the implementation and semantics would both be vastly different. For ui.c, treating internal callback as RemoteUI doesn't make any sense (none of the code there is needed for internal callback), having separate LuiUI type to implement the same interface would be much simpler.

In this module, we could code golf away the separate array by storing callbacks as negative numbers or something. But it wouldn't make callbacks to be channels, just storing a union of both types in the same array.

justinmk · 2019-04-28T15:11:55Z

LGTM except for comment. After basic test coverage let's merge this and build on it.

bfredl · 2019-06-02T18:57:20Z

Tried a different design that IMHO seems easier to use: the caller will supply separate on_lines and on_changedtick callbacks. So it is easy to ignore changedtick events if they are irrelevant. It is still possible to specify the same callback for both, in analogy with jobstart(). Also, I hope to implement a more detailed, byte-based update (which would have less impedance missmatch with tree-sitter) as a follow up, which for lua could use a different callback name.

runtime/doc/api.txt

src/nvim/buffer_defs.h

bfredl · 2019-06-04T12:56:18Z

Merged, further improvements will be done in follow-up PR:s, both to the lua callbacks in general and buf_attach in particular (byte based deltas etc).

This release represents ~2700 commits since v0.3.4, the previous non-maintenance release. Besides the highlights listed below, this release features vast improvements to documentation, internal subsystems and test/CI infrastructure, and 700+ patches merged from Vim. FEATURES: New API functions: nvim_create_buf: create various kinds of buffers nvim_get_context, nvim_load_context 8e6b0a7 #10619 API: Context: save/restore/inspect editor state nvim_input_mouse: perform mouse actions nvim_open_win: create floating windows (and external, for supporting UIs) nvim_paste: paste text at cursor nvim_put: put text at cursor nvim_select_popupmenu_item: perform popupmenu actions nvim_set_keymap: create/delete mappings nvim_set_vvar: set v: variables nvim_ui_pum_set_height nvim_ui_try_resize_grid nvim_win_close: close windows nvim_win_get_config: get window configuration nvim_win_set_config: reconfigure windows New UI events: redraw.grid_destroy redraw.hl_group_set 8a3f858 #10504 UI/highlight: expose builtin highlight groups using hl_group_set event redraw.msg_clear redraw.msg_history_show redraw.msg_ruler redraw.msg_set_pos redraw.msg_show redraw.msg_showcmd redraw.msg_showmode redraw.win_close redraw.win_external_pos redraw.win_float_pos redraw.win_hide redraw.win_pos API f5c56f0 #9170 API/Lua: nvim_buf_attach: support Lua callback 82d48c0 #9896 API: emit nvim_error_event on failed async request b9ad12e #9992 UI/nvim_ui_attach(): add `override` option 3d1ed7c #9993 UI/ext_messages: learn more message kinds 8ed54bb #9547 proper multiline error message for rpcrequest, API wrappers Lua This release introduces "Nvim-Lua standard library". See ":help lua-stdlib". 89d7e24 #9463 Lua stdlib: vim.inspect, string functions 8e941c5 #9740 Lua: generate documentation from docstrings 1cbe014 #9301 lua/stdlib: Introduce vim.shared c83926c #10123 Lua: introduce vim.loop (expose libuv event-loop) 81e1dbc #10120 Lua: vim.schedule(cb) 1f54f68 #10688 Lua: minimal UTF-16 support needed for LSP 6fb0020 #10513 Lua encoding support c0993ed Lua: support getting UTF-32 and UTF-16 sizes of replaced text b0e2619 Lua: add {old_byte_size} to on_lines buffer change event UI: - The Nvim 0.3.4 UI protocol introduced line-based updates instead of legacy char-based updates. Nvim 0.4 continues to evolve the UI protocol. See ":help ui". Legacy UI clients are supported. See ":help api-contract". 9a1675b #6619 Floating windows - Can be (re)positioned, anchored, external. - Are real windows showing real buffers. No shortcuts, hacks, or compromises. - Support all features and API of normal windows, plus more. 6427894 #8455 Multigrid: "windows drawn on separate grids" - Windows are logically isolated internally. - Windows are sent to UIs as distinct objects, so that UIs can control layout instead of being stuck with the classic TUI layout. - Per-window font-size, dimenions, line-spacing. - Compositor: Internal subsystem for composing grids. 3855204 #6917 UIEnter, UILeave 788bcbb #9923 ui: ":syn blend=", 'winblend' 7cf7c0a #9575 ui: 'redrawdebug' option for flexible debugging of redrawing 5c836d2 #9607 wildoptions=pum (enabled by default) 37f8df8 #9571 UI: 'pumblend' option for semi-transparent popupmenu c403a95 #9446 Visual: highlight char-at-cursor - Traditionally Vim's visual selection does "reverse mode", which perhaps conflicts with the non-blinking block cursor. But 'guicursor' defaults to a vertical bar for selection=exclusive, and this confuses users who expect to see the text highlighted. :terminal fc27dc9 #8550 autocmds: TermEnter, TermLeave d13803f #9810 keymap, terminal: more keycodes 3b56f59 #9535 :terminal : Fix F1-F4 key codes 2d4a37e #10370 :ls : show "R", "F" for terminal-jobs fd0fd75 #9966 terminal: swap priority of terminal, editor highlights 7bb858c #9494 libvterm 0.1 TUI 3afb397 syntax, TUI: support "strikethrough" ccbcd39 #9408 TUI: "title stacking" unconditionally 298608f #9509 TUI: detect background color, set bg=dark/light 42f492a #9097 TUI: handle Smulx extension capability (extended underline) 424ddd0 #10205 TUI: support rgba background detection 9b43832 #9601 TUI: italics in tmux, Terminal.app f6fb370 #9793 keymap: support more (keypad) keycodes 3340e08 #9423 TUI: Konsole DECSCUSR fixup :checkhealth d0fd66b health/provider.vim: check curl HTTPS support c38862a #10490 checkhealth: try yarn if npm is missing 43356a4 #9929 health: check if tmux enabled true colors ec5a4d8 #9548 checkhealth: validate locale providers (clipboard, python, etc.): 96be8a2 #10161 Allow reloading providers (useful for UIs/clients) db3c797 #9487 provider: improve error message if provider is missing Various: 36762a0 #9295 signs: support multiple columns 801fe79 #10382 eval: wait() (wait for any condition) 9df3a67 #10400 MsgArea highlight; message grid a9bea8c #10790 keymap: allow modifiers to multibyte chars, like <m-ä> 25e0a44 #10878 #4448 paste: redesign (10x+ faster pasting; extensible vim.paste Lua hook) ef5037e #9706 autocmd: introduce "++once" feature 175398f #9616 add CompleteChanged autocmd 7fcf2f9 #9717 TextYankPost: add v:event["inclusive"] 3a699a7 #8364 termdebug.vim plugin ca1ce59 #9709 performance: use os_copy to create backups ed0e96c man.vim: set 'linebreak' 70f6939 #9564 events: add "Signal" event f89d0d8 #9568 inccommand: auto-disable if folding is slow FIXES: 41bb68b #10584 process_stop: uv: do not close stdin first/explicitly e50aa2a #10117 normal: Don't exit CTRL-O mode after processing K_EVENT 95fa71c #9504 :recover : Fix crash on non-existent *.swp 5a836d4 #9507 screen: don't unconditionally clear messages on window scroll 149dcbf #10021 channel: refactor events, prevent recursive invocation of events d19ff73 #10107 Fix multiple c_CTRL-D showing statusline b65a7b7 #10103 Fix wildmode=list,full and display+=msgsep interaction 0be6d3c #9634 fsync: Ignore ENOTSUP. Fix writing to SMB. b247c6f #10025 kbtree: pointer UB and unitialized value fixes 018e0d5 #9643 API/buffer-updates: always detach on buf-reload 400ee59 #9961 API: fix cursor position when lines are added 769f44e #9911 win/defaults: Use "…/nvim-data/site" in 'runtimepath' 83d5716 #9911 spellfile.vim: store files in stdpath('data') 8dbf231 #9887 RPC: conform message-id type to msgpack-RPC spec 5f996e3 #9894 options: properly reset directories on 'autochdir' 4c4a570 #9807 various CursorMoved fixes 943bedf #9853 event-loop: do not set CA_COMMAND_BUSY 9d207fd #9693 dictwatcheradd(): support b:changedtick 2d50bf3 #9789 mac: fix locale detection c563133 #9754 :mksession : restore tab-local working directories 092e7e6 #9703 #9703 executable(): return false if user is not owner 11a481f #9686 env var fixes/improvements 8e54847 #9666 #7920 os/env: Fix completion of multibyte env var names 5193826 #10468 Fix is_executable_in_path() on Windows 8eaa452 #9516 win: exepath(), executable() fixes f55c1e4 #10544 reltimefloat(): allow negative result b08dc3e #10561 win: jobstart(), system(): $PATHEXT-resolve exe 7cc2b72 #10392 TextYankPost: spurious/too-early dispatch during delete 6e01ed6 OpenBSD: stop jobs/processes properly 58dd5fc #10522 jobstop(): close channel before process_stop() 8363202 #10959 improved resize behavior (all UIs) c6eb1f4 #10830 API: fix nvim_command_output buffer overflow cbfd18c #10763 startup: handle 'guicursor' after user config b8f2436 #10915 jobwait(): fix race if job exits quickly 2fafed6 #10765 clipboard: handle/avoid SIGTERM with previous owner 8aca932 #9954 clipboard: setreg("*") with clipboard=unnamed 3f10c5b #9480 performance: clipboard/macOS: assume that pbcopy works 48efafc #10398 screen: disable redrawing inside VimResized 5e4b93a #10389 API/Lua: make nvim_execute_lua use native lua floats, not special tables 8c6f5b7 #9934 Spurious quote mark in command line when typing <C-R> a8a38f3 Lua 5.2/5.3 compat :terminal 47b7b47 #10700 :terminal : update buffer when switching tabpage 5225c1e #9605 terminal: Fix potential invalid local 'scrollback' 894f6be #8325 :terminal : set topline based on window height 8171e96 #9551 Improve :terminal resize d928b03 #9856 :stopinsert should leave terminal-mode 3f71218 #9926 :terminal : fix: Using `:stopinsert` while in normal mode 5020daa #9883 ui/terminal: make terminal state redraw like any other state TUI: 9f19e8d #9443 TUI: Do not disable BCE for builtin terminfos a4076e5 #9474 win/TUI: fix text overrides line numbers 533d4a3 #9645 TUI: do not resize host-terminal on startup b51e5d8 #9688 tui_tk_ti_getstr: handle weird value 1f5eac1 #10785 TUI: fix data-race during resize CHANGES: 9697c7f #8194 fix menu_get() 7f2e43c #9520 improve Lua error messages c234318 #9526 Remove jemalloc baf93d9 #9581 UI: always use concrete colors for default_colors_set 91688b4 #9563 defaults: set 'scrollback' to -1 by default bb24fec #10136 defaults: exclude "S" from 'shortmess' ddd0eb6 #8540 startup: -es/-Es (silent/batch mode): skip swapfile 3536249 #9805 jumplist: avoid extra tail entry 939d905 #10573 channels: reflect exit due to signals in exit status code 45c34bd #10689 :doautocmd : Never show "No matching autocommands" fb19aee #9110 API: make nvim_win_set_option() set window-global, not buffer-local abfc8b3 #10778 emsg_multiline: log Vim errors 06d9cc7 #10657 exists("$FOO"): return false for empty env var 6616d1d #10743 win/env: Vim-compat: Empty string deletes env var 7d66483 #10662 win: expand nested env var in $HOME 2816bc8 #8349 edit.c: Disable indent during completion 58f505d #9829 startup: remove TUI init special-case Historically Vim/Nvim does backflips to handle input and show messages before a UI is available. This logical contradiction was already fixed for remote UIs (#9024 c236e80). Fixing it also for the TUI avoids problems on Windows, simplifies the logic, and avoids races like #9959.

marvim added the WIP label Oct 29, 2018

This was referenced Nov 2, 2018

structured/"rich" text, text annotations/overlay #1767

Closed

[WIP] Tree-sitter integration #9219

Closed

justinmk reviewed Apr 28, 2019

View reviewed changes

bfredl force-pushed the lua_cb branch 2 times, most recently from 6c9cc36 to 5178452 Compare June 2, 2019 18:50

justinmk reviewed Jun 2, 2019

View reviewed changes

runtime/doc/api.txt Outdated Show resolved Hide resolved

bfredl force-pushed the lua_cb branch from 5178452 to e2b5e7e Compare June 2, 2019 20:46

justinmk reviewed Jun 2, 2019

View reviewed changes

src/nvim/buffer_defs.h Outdated Show resolved Hide resolved

bfredl force-pushed the lua_cb branch 2 times, most recently from 32e8e8e to 99093ff Compare June 3, 2019 17:46

bfredl mentioned this pull request Jun 3, 2019

[WIP] expose the libuv event loop in lua with luv bindings #9546

Closed

bfredl force-pushed the lua_cb branch 2 times, most recently from 0d39d34 to 7031da2 Compare June 4, 2019 11:05

api: allow nvim_buf_attach from lua using callbacks

f5c56f0

bfredl force-pushed the lua_cb branch from 7031da2 to f5c56f0 Compare June 4, 2019 11:48

bfredl changed the title ~~[WIP] lua callbacks for nvim_buf_attach (and others?)~~ lua callbacks for nvim_buf_attach Jun 4, 2019

bfredl merged commit 3adb8a1 into neovim:master Jun 4, 2019

justinmk added api libnvim, Nvim RPC API lua stdlib labels Jun 4, 2019

KillTheMule mentioned this pull request Jul 8, 2019

[RFC] Built-in LSP Support #10222

Closed

20 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

lua callbacks for nvim_buf_attach #9170

lua callbacks for nvim_buf_attach #9170

Uh oh!

bfredl commented Oct 29, 2018

Uh oh!

KillTheMule commented Jan 24, 2019

Uh oh!

bfredl commented Jan 24, 2019

Uh oh!

KillTheMule commented Jan 24, 2019

Uh oh!

bfredl commented Jan 24, 2019

Uh oh!

justinmk Apr 28, 2019

Uh oh!

bfredl May 28, 2019

Uh oh!

justinmk commented Apr 28, 2019

Uh oh!

bfredl commented Jun 2, 2019

Uh oh!

Uh oh!

Uh oh!

bfredl commented Jun 4, 2019

Uh oh!

Uh oh!

Uh oh!

lua callbacks for nvim_buf_attach #9170

lua callbacks for nvim_buf_attach #9170

Uh oh!

Conversation

bfredl commented Oct 29, 2018

Uh oh!

KillTheMule commented Jan 24, 2019

Uh oh!

bfredl commented Jan 24, 2019

Uh oh!

KillTheMule commented Jan 24, 2019

Uh oh!

bfredl commented Jan 24, 2019

Uh oh!

justinmk Apr 28, 2019

Choose a reason for hiding this comment

Uh oh!

bfredl May 28, 2019

Choose a reason for hiding this comment

Uh oh!

justinmk commented Apr 28, 2019

Uh oh!

bfredl commented Jun 2, 2019

Uh oh!

Uh oh!

Uh oh!

bfredl commented Jun 4, 2019

Uh oh!

Uh oh!