Also change I would make, but in a separate PR: move almost all job/channel code from eval.c to channel.c, also expose functionality natively in lua, so lua functions/callbacks can use lua binary strings instead of linelists, also rpc functions can do directly lua<->msgpack (primarily for semantics/convenience, only secondarily for performance).

lua callbacks could even be run more asynchronously than vimL callbacks (i e in situations where vimL is forbidden) and must use vim.schedule(callback) for anything that touches vimL or mutates editor state. I'm not 100% we want this, but if we do, we should introduce this early on.

Lua is no more thread-safe then VimL is, except that you can asynchronously use different lua_States and VimL has a bunch of globals scattered throught the code instead.

I'm not thinking of threads, just event handling in the main thread. There are many circumstances where the uv event loop is being run, but most events (that can execute arbitrary vimscript, most API functions) are blocked. We could be more liberal with executing lua callbacks if this behavior is introduced from start. Alternatively they can by default behave like vimL callbacks, and this behavior behind a flag.

Now allows byte based sockets:

func! SockHandler(chan_id, data, event)
   let g:event = [a:chan_id, a:data, a:event]
endfunc

let x = sockconnect('pipe', '/some/nvim/socket`, {'on_output': 'SockHandler'})
call jobsend(x, "blargh")
echo g:event 

should see something like
[4, ['~@l^A~@1~@^A~@^A~@^A~@^A~@^A~@^A~@^A~@^A~@n^A~@IMessage is not an array~@@'], 'socket']

Also implement stdioopen({'rpc':..., 'on_stdin':...}). Only works in --headless mode, obviously. nvim --embed is then essentially a short-hand for nvim --headless --cmd "call stdioopen({'rpc': v:true})". Also stdio channel is guaranteed to be 1, so tests can just use 1 instead of needing to query nvim_get_api_info

Rather than allowing a separate close callback on_stdout: {"data": "Callback1", "close": "Callback2"}maybe be should just indicate this with special value, like data set to [] in the same callback. But it might be a clumsy with all-data-on-close behavior. The best It would also be good to distinguish signalling EOF in PTY input, maybe this can be data=[''].

Another solution is to rename the on_stdout/on_stderr argument, so if the new name is used, another flags argument will be received, but the old name gives compatible behavior.

Implemented option stdout_buffered to get all output data at once (at EOF).

Now jobclose(chan, part) works for all channels. Obviously it should be renamed, just like jobsend(). Currently we have

Functions that return channels

jobstart()
termopen()
sockconnect()
stdioopen()

Functions that take jobs

jobpid()
jobresize()
jobstop()
jobwait()

Functions that take any channel

rpcrequest()
rpcnotify()
jobsend()
jobclose()

(ignoring deprecated functions that can be done using any above)

What is also missing is able to write to nvim's own stderr. It should probably not be mixed into the stdio channel as chansend(1, datas, "stderr") would be quite ugly, and also stderr is always possible even if stdin/out is not used as a channel. Rather, either a dedicated channel chansend(2, data) or a specific function write_stderr(data).

merged. The "channels can connect to socket" (bytes mode) test was a bit flaky the other day, but has been fine the latest retries. If it causes trouble again I will mark it as pending (and eventually fix it by adding a proper bytes server shim, instead of abusing msgpackdump for this purpose).

This seems breaking old behavior... An extra newline seems to be added. I'll investigate deeply later.

https://travis-ci.org/lambdalisue/gina.vim/jobs/308500736#L2077
https://ci.appveyor.com/project/lambdalisue/gina-vim/build/655/job/brq82xko9xudoi8p#L115

Hmm, I first though it was issue of the extra EOF event that is added. But gina#util#extend_content() looks like it should handle it perfectly fine. ((unrelated note: you can avoid the extra branch by initializing job.stdout/stderr with [''] instead of []) Can you confirm it is what's been used in this test and not some other stuff, in vital for instance?

https://travis-ci.org/lambdalisue/gina.vim/builds/308500727
https://ci.appveyor.com/project/lambdalisue/gina-vim/build/655

Only the neovim HEAD shows fail so I think test is correct.

I first though it was issue of the extra EOF event that is added.

Do you have detail? I would like to investigate that.

Can you confirm it is what's been used in this test and not some other stuff, in vital for instance?

Possible. I'll check thanks 👍

https://github.com/lambdalisue/gina.vim/blob/master/test/gina/process.vimspec#L31

You were correct. The test has its own inplementation which just add the data to a list.

So is this behavior change intentional? And is there any documentation about behavior change?

@lambdalisue See "Note 2" in :help job_control. The behavior didn't change, it may have worked by chance before.

@justinmk The behavior did change so that EOF is identifiable, but it should be purely an information gain change. 90 % of plugins that uses the recommended pattern should notice no breaking change, 9% was broken already but extremely unlikely seen because flushing behavior is kind of deterministic even if no guarantees actually was made.1% is the "spacebar heating" cases we need to give recommendations individually.

90 % of plugins that uses the recommended pattern should notice no breaking change

so what is the recommanded pattern ?

if a shell command output in terminal is:

foo

foo
foo

then the a:data of stdout func is ['foo', '', 'foo', 'foo']?

and any document show the differences, or any job example show the differences?

90 % of plugins that uses the recommended pattern should notice no breaking change

In my case, I was using a low-level implementation (only in one test, other parts I was using a recommended pattern) so the test had failed.
Once I changed the implementation as an example in a help, the test pass 🎉

lambdalisue/vim-gina@0bdb844

@wsdjeg

Prob "Note 2" in :help job_control is the recommended pattern.

@lambdalisue Thanks, I got it.

@bfredl is there a check flag for this change? just to make sure my config is compatibility to old version of neovim, for example

   " some new func added in this pr
   if exist('*new_func_name')
       " old stdout handle func
   else
      " new fixed stdout handle func
   endif

The point is that a function that parses lines correctly, shouldn't need to change behavior depending on version. But you can of course check for chansend() existing if really want.

Are there any plans to extend sockconnect to work without the RPC flag? I found a use-case for such a feature while working with vim-picker.

@boxofrox This is already implemented, but only on master.