paste: redesign, fixes, 10x speedup #4448

justinmk · 2016-03-14T00:48:42Z

This PR improves paste behavior and performance.

Performance: 10x-100x faster (try pasting MB of input, or paste a lot of text with filetype=cpp or filetype=ruby)
Extensibility: users can control paste behavior by redefining vim.paste (Lua)
API: nvim_paste for UIs/clients
API: nvim_put inserts text (weird but true: this was difficult before)
Fixes:

Tasks:

implement as callback instead of pre/post autocmds
Indicate paste start/stop as parameter to handler?
generalize string_to_array() ?
API: nvim_put [WIP] nvim_put #6819
API: nvim_paste
redo (dot-repeat)
'nomodifiable' buffer
documentation

TODO (future):

Replace/Operator-pending/Visual modes
remove 'pastetoggle' p_pt
Dot-repeat is slow (same behavior in Vim) because it is subject to Vim auto-indent/etc.
Mechanism for associating a callback with a nvim_input chunk. API: nvim_input_flush #10826
TextPutPre/TextPutPost (analogs for TextYankPost) ?
Send paste sequences to :terminal
- As @ZyX-I mentioned this would also be warranted for p, :put, etc.
handle missing "stop paste" code (how? timeout...?) paste: handle "dangling paste" (missing stop-paste code) #10865

runtime/doc/eval.txt

@@ -3435,11 +3435,11 @@ getchar([expr])						*getchar()*
 			:endfunction
 <
 		You may also receive synthetic characters, such as
-		|<CursorHold>|. Often you will want to ignore this and get
+		|<LeftMouse>|. Often you will want to ignore this and get


tarruda · 2016-03-14T11:17:52Z

@justinmk I'm not sure I understand the goal behind this PR, is it to decouple the logic of automatically toggling paste with a special key?

If so, I don't see much gain here: We still have bracketed paste parsing hardcoded into the TUI, only the set paste logic will be moved to vimscript(even so, what would be the purpose of allowing autocmds to be attached to paste?). But I do see a worthly goal in this:

I am wondering if it would be a good idea to provide some internal generic mechanism for placing K_EVENT in the input stream and associating it with a callback.

If you could implement this, then yes, we could completely decouple bracketed paste(and possibly many other terminal sequences) from the tui. But it should not be K_EVENT, since it is already used for another purpose(and no, we can't place arbitrary events so they are processed at specific times during input processing).

However, this seems to overlap with @bfredl's work on #4419 :

map <command> <Paste> if &paste | set nopaste | else | set paste

Maybe it would be even better if we simply supported user-defined synthetic keys:

map <command> <User:SOME_KEY_ID> if &paste | set nopaste | else | set paste

The question is: How can a plugin tell the UI to put such synthetic keys into the input buffer? Right now we can't, but the TUI will soon allow configuration, so a bracketed paste plugin could be summarized to something like this:

let g:tui_keys_override = {
\  '\x1b[200~': '<User:Paste>'
\  '\x1b[201~': '<User:Paste>'
\}
imap <command> <User:Paste> if &paste | set nopaste | else | set paste

That is, the plugin asks the tui to translate bracketed pastes into <user:paste>, and maps it to a code that toggles the paste option.

ZyX-I · 2016-03-14T12:22:49Z

src/nvim/main.c

@@ -328,6 +326,9 @@ int main(int argc, char **argv)
                 "'\\c\\m" PROTO "\\(.\\{-}\\)//'), 1, '')})");
 #undef PROTO

+  do_cmdline_cmd("autocmd PastePre  * set paste|if mode()!~#'c\\|t'|call feedkeys('\x1C\xEi','n')|endif");
+  do_cmdline_cmd("autocmd PastePost * set nopaste|if mode()!~#'c\\|t'|call feedkeys('\x1C\xE','n')|endif");


\xE looks weird, better write \x0E.

Though no, it is better to write this as \"\\<C-\\>\\<C-n>\".

And I would also suggest to create autocmd group __Builtin for all built-in autocommands (currently only this one and term://* BufReadCmd), so that they are easy to identify, making user aware of what he sees and, possibly, changes.

Regex is better written as stridx('ct', mode()) == -1 or mode() !~# '[ct]'.

ZyX-I · 2016-03-14T16:25:46Z

@justinmk Bracketed paste is terminal-UI-specific thing. GUIs do not have anything like this, all they have are events from user: clipboard and mouse. There is no “external paste”, there are things like "+p, :put and events which map to this. In any case clipboard contents is received in one piece, it is not a stream like {paste start}{contents}{paste end}.

ZyX-I · 2016-03-14T16:29:49Z

@justinmk <MiddleMouse>, <C-v>, etc are handled by the application itself and it is UI which initiates pasting upon receiving an appropriate event. There are conventions and toolkits which follow conventions by default, but they are not enforced and toolkits can be told to not follow them as well. With bracketed paste it is terminal which initiates pasting, which is an “external” application.

ZyX-I · 2016-03-14T17:06:11Z

By the way, consider that user pressed <S-Insert>foo, which translated into {paste start}{inserted contents}{paste end}foo. Can I use Paste* events to get {inserted contents} without foo?

Also I saw that you test ignoring second {paste start} in {paste start}{paste start}. This is incorrect, I checked konsole using

echo $'\e[200~' | xclip -i
echo $'\e[?2004h' ; stty -echo ; cat -v

(with <MiddleMouse> after last <CR>) and got

^[[200~^[[200~
^[[201~

: no escaping. Spurious {paste start} should be part of the pasted text, not ignored. Double {paste end}/{paste end} without {paste start} should be ignored though I think.

justinmk · 2016-03-14T17:10:36Z

Can I use Paste* events to get {inserted contents} without foo?

The text is not provided in any register or v: variable, but the buffer contents at time of PastePost should not contain foo. And you could compare this to buffer contents at time of PastePre.

Spurious {paste start} should be part of the pasted text, not ignored. Double {paste end}/{paste end} without {paste start} should be ignored though I think.

Thanks, I will fix it.

tarruda · 2016-03-14T17:10:48Z

I don't think so. The toggle does not work. It's broken. The system needs to know "this is the start" and "this is the end". The state of 'paste' cannot be a factor in this, the user may already have it enabled.

The old code was sending to toggle on and another one later in the stream to toggle off. This is the cause of bugs we've had for 6 months.

Alright. What about your suggestion of adding a generic mechanism to allow execution of vimscript when a certain synthetic keys are processed? Assuming the TUI allows the user to override terminal code parsing(it will), a bracketed paste plugin could do this:

let g:tui_keys_override = {
\  '\x1b[200~': '<User:PasteEnable>'
\  '\x1b[201~': '<User:PasteDisable>'
\}

It seems like something that could work. Even FocusGain/FocusLost could work nicely on such infrastructure.

ZyX-I · 2016-03-14T17:11:38Z

I also do not see tests for pasting while in all other modes (visual, select, operator-pending, replace, virtual replace).

tarruda · 2016-03-14T17:17:19Z

Adapting my previous comment to @bfredl's idea, here's how bracketed paste would be implemented as a plugin:

let g:tui_keys_override = {
\  '\x1b[200~': '<Cmd>set paste<cr>'
\  '\x1b[201~': '<Cmd>set nopaste<cr>'
\}

justinmk · 2016-03-14T17:20:26Z

Assuming the TUI allows the user to override terminal code parsing(it will), a bracketed paste plugin could do this:
let g:tui_keys_override

I suppose that would be fine if no one is convinced that PastePre/PastePost will be useful for GUIs (I gave use-cases above). But:

It will block this fix for a long time. Paste has been broken for more than 6 months.
It means we're back to asking users to fiddle with termcodes--even when the work was already done to abstract it for certain cases.
It won't apply to GUIs, so configuration of this behavior won't be consistent across UIs.

bfredl · 2016-03-14T17:20:27Z

Hmm I wonder how to implement "Send paste sequences to :terminal" with that. Is the remapping done by the tui module or do you mean runtime will add those mappings to insert mode?

ZyX-I · 2016-03-14T17:24:45Z

@bfredl Neither of suggested variants allow implementing “send paste sequences to :terminal”, because this needs hooking "*p. Though bracket pasting to terminal may be implemented based on PastePre/PastePost, this also needs getting terminal state.

This is "readfile()-style", see also ":help channel-lines".

Fixes strange behavior where sometimes the buffer contents of a series of paste chunks (vim._paste) would be out-of-order. Now the tui_spec.lua screen-tests are much more reliable. But they still sometimes fail because of off-by-one cursor (caused by "typeahead race" resulting in wrong mode; fixed later in this patch-series).

- Send `phase` parameter to the paste handler. - Redraw at intervals and when paste terminates. - Show "..." throbber during paste to indicate activity.

Workaround this failure: [ ERROR ] test/functional/terminal/tui_spec.lua @ 192: TUI paste: exactly 64 bytes test/functional/helpers.lua:403: retry() attempts: 478 test/functional/terminal/tui_spec.lua:201: Expected objects to be the same. Passed in: (table: 0x47cd77e8) { *[1] = 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz endz' } Expected: (table: 0x47cd7830) { *[1] = 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz end' } This happens because `curwin->w_cursor.col` is sometimes decremented at the end of `do_put`... because the editor is in Normal-mode instead of the expected Insert-mode. Caused by "typeahead race" (neovim#10826): there may be queued input in the main thread not yet processed, thus the editor mode (`State` global) will be "wrong" during paste. Example: input "i" followed immediately by a paste sequence: i<start-paste>...<stop-paste> ^ "i" does not get processed in time, so the editor is in Normal-mode instead of Insert-mode while handling the paste. Attempted workarounds: - vim.api.nvim_feedkeys('','x',false) in vim._paste() - exec_normal() in tinput_wait_enqueue() - LOOP_PROCESS_EVENTS(&main_loop,…,0) in tinput_wait_enqueue() ref neovim#10826

- Normal-mode redo idiom(?): prepend "i" and append ESC. - Insert-mode only needs AppendToRedobuffLit(). - Cmdline-mode: only paste the first line.

HACK: The cursor does not get repositioned after the paste completes. Scheduling a dummy event seems to fix it. Test case: 0. Revert this commit. 1. Paste some text in Normal-mode. 2. Notice the cursor is still in the cmdline area.

- Show error only once per "paste stream". - Drain remaining chunks until phase=3. - Lay groundwork for "cancel". - Constrain semantics of "cancel" to mean "client must stop"; it is unrelated to presence of error(s).

- nvim_paste(): Marshal through luaeval() instead of nvim_execute_lua() because the latter seems to hide some errors. - Handle 'nomodifiable' in `nvim_put()` explicitly. - Require explicit `false` from `vim.paste()` in order to "cancel", otherwise assume true ("continue").

justinmk · 2019-08-27T21:41:50Z

Resolution/Summary

Thanks to @ZyX-I for review and suggestions.
Remaining items will go in followup PR.
Marking these vim-patches as NA (cc @janlazo):

vim-patch:8.1.1198
vim-patch:8.1.0224
vim-patch:8.0.1299
vim-patch:8.0.0569
vim-patch:8.0.0303
vim-patch:8.0.0296
vim-patch:8.0.0244
vim-patch:8.0.0238
vim-patch:8.0.0232
vim-patch:8.0.0231
vim-patch:8.0.0230
vim-patch:8.0.0210

Notes:

cursor is hidden after paste completes. Workaround: send dummy event 4344ac1:
```
loop_schedule_deferred(&main_loop, event_create(x_dummy_event, 0));
```

Locked to keep the summary visible. You can always chat or open a ticket if you have new information/topics to discuss.

- Introduce TRY_WRAP() until we have an *architectural* solution. - TODO: bfredl idea: prepare error-handling at "top level" (nv_event). - nvim_paste(): Revert luaeval() hack (see parent commit). - With TRY_WRAP() in nvim_put(), 'nomodifiable' error now correctly "bubbles up".

marvim added the WIP label Mar 14, 2016

justinmk force-pushed the paste branch 2 times, most recently from c9b85ab to 8898ee5 Compare March 14, 2016 03:41

justinmk mentioned this pull request Mar 14, 2016

[RFC] [#3476] Add bracketedpaste option #3906

Closed

justinmk reviewed Mar 14, 2016
View reviewed changes

tarruda mentioned this pull request Mar 14, 2016

[RFC] implement <Cmd> key #4419

Merged

ZyX-I reviewed Mar 14, 2016
View reviewed changes

This comment has been minimized.

Sign in to view

justinmk added 10 commits August 27, 2019 21:19

API: nvim_put neovim#6819

e1177be

paste: use nvim_put()

5a2894d

paste: test

d303790

paste: edge-case: handle EOL at end-of-buffer

0221a92

This is "readfile()-style", see also ":help channel-lines".

API: nvim_put: Avoid "N more lines" message

517bf99

test/tui_spec: connect to child session

1fdae25

API: nvim_put: "follow" parameter

93e5f02

paste: phases, dots

5ae6849

- Send `phase` parameter to the paste handler. - Redraw at intervals and when paste terminates. - Show "..." throbber during paste to indicate activity.

justinmk force-pushed the paste branch from 7dc335b to ce155b4 Compare August 27, 2019 19:19

justinmk added 6 commits August 27, 2019 22:13

API: nvim_paste

eacc70f

paste: insert text "before" cursor in Insert-mode

bfc5a18

paste: implement redo (AKA dot-repeat)

5b41070

- Normal-mode redo idiom(?): prepend "i" and append ESC. - Insert-mode only needs AppendToRedobuffLit(). - Cmdline-mode: only paste the first line.

paste: tickle cursor

4344ac1

HACK: The cursor does not get repositioned after the paste completes. Scheduling a dummy event seems to fix it. Test case: 0. Revert this commit. 1. Paste some text in Normal-mode. 2. Notice the cursor is still in the cmdline area.

paste: handle vim.paste() failure

ed60015

- Show error only once per "paste stream". - Drain remaining chunks until phase=3. - Lay groundwork for "cancel". - Constrain semantics of "cancel" to mean "client must stop"; it is unrelated to presence of error(s).

paste: make vim.paste() "public"

87389c6

justinmk force-pushed the paste branch from ce155b4 to 6c44a9d Compare August 27, 2019 20:15

justinmk mentioned this pull request Aug 27, 2019

paste: handle "dangling paste" (missing stop-paste code) #10865

Open

justinmk force-pushed the paste branch from 34d1989 to 2fd3b6c Compare August 27, 2019 21:37

justinmk force-pushed the paste branch 2 times, most recently from 99a1753 to fee2e1c Compare August 27, 2019 22:13

justinmk force-pushed the paste branch from fee2e1c to 3157bae Compare August 27, 2019 22:57

justinmk merged commit 82d52b2 into neovim:master Aug 27, 2019

justinmk deleted the paste branch August 27, 2019 23:56

neovim locked as resolved and limited conversation to collaborators Aug 27, 2019

justinmk added the performance performance, latency, cpu/memory usage label Sep 5, 2024

Uh oh!

paste: redesign, fixes, 10x speedup #4448

paste: redesign, fixes, 10x speedup #4448

Uh oh!

Conversation

justinmk commented Mar 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

tarruda commented Mar 14, 2016

Uh oh!

ZyX-I Mar 14, 2016

Choose a reason for hiding this comment

Uh oh!

ZyX-I Mar 14, 2016

Choose a reason for hiding this comment

Uh oh!

ZyX-I Mar 14, 2016

Choose a reason for hiding this comment

Uh oh!

ZyX-I Mar 14, 2016

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ZyX-I commented Mar 14, 2016

Uh oh!

ZyX-I commented Mar 14, 2016

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ZyX-I commented Mar 14, 2016

Uh oh!

justinmk commented Mar 14, 2016

Uh oh!

tarruda commented Mar 14, 2016

Uh oh!

ZyX-I commented Mar 14, 2016

Uh oh!

tarruda commented Mar 14, 2016

Uh oh!

justinmk commented Mar 14, 2016

Uh oh!

bfredl commented Mar 14, 2016

Uh oh!

ZyX-I commented Mar 14, 2016

Uh oh!

justinmk commented Aug 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Resolution/Summary

Uh oh!

Uh oh!

justinmk commented Mar 14, 2016 •

edited

Loading

justinmk commented Aug 27, 2019 •

edited

Loading