Regarding this change (github seems to be confused if I comment inline):

+  // See https://gist.github.com/XVilka/8346728 for more about this.
+  if (putty || xterm || rxvt || linuxvt || konsole || iterm || truecolor) {
+    data->unibi_ext.set_rgb_foreground = (int)unibi_add_ext_str(ut, NULL,
+        "\x1b[38;2;%p1%d;%p2%d;%p3%dm");

Previously this was set always and the nvim option 'termguicolors' controlled it. Is it possible that some truecolor-capable terminals will be missed now? What do we gain by leaving this undefined in the false case of this condition?

Related issue:
Shougo/denite.nvim#280

@tmccombs Could you test the branch?

I hope that you were falling foul of the "everything that is not libvte supports DECSCUSR" test in the old code, @Shougo and @tmccombs. If you still have a problem after switching to this code, I am interested, in particular in what terminal or terminal emulator you see the problem on and what your TERM environment variable is.

Note that the design here is very much to make any further problem a terminfo problem, rather than a nvim-specific problem. Fix what the terminfo database says for problem cases, and both nvim and tmux benefit. But also there is a way to avoid the terminfo fixup that nvim does, if you actually need to. (I hope and suspect that you will not.) nvim does the fixup when the Ss extended terminfo capability is absent. If it is present but an empty string nvim will simply do what terminfo says to do.

That's your local fix should you continue to experience letters q on your screen with this code in place. As I said, I suspect that it will not be necessary, and that the old libvte check was the source of your problem.

See #6793 for discussion of the QuickBuild failure.

I have tested the branch.
Wow! The problem is fixed!

@Shougo works for me. Nice work @jdebp!

@jdebp The travis CI failure looks legitimate, or the test needs to be updated: empty TERM appears to result in t_Co=8.

Note that we only support t_Co for script compatibility. If you intentionally want to report t_Co=8 when TERM is empty, that's fine, just need to update that test.

[1m�[31m[  ERROR   ]�[0m�[0m �[36m...uild/neovim/neovim/test/functional/terminal/tui_spec.lua�[0m @ �[36m351�[0m: �[1mtui 't_Co' (terminal colors) unknown TERM sets empty 't_Co'�[0m
./test/functional/ui/screen.lua:301: Row 2 did not match.
Expected:
  |{1: }                                                 |
  |*{4:~                                                 }|
  |{4:~                                                 }|
  |{4:~                                                 }|
  |{5:[No Name]                                         }|
  |                                                  |
  |{3:-- TERMINAL --}                                    |
Actual:
  |{1: }                                                 |
  |*~                                                 |
  |~                                                 |
  |~                                                 |
  |{5:[No Name]                                         }|
  |8                                                 |
  |{3:-- TERMINAL --}                                    |

The quickbuild failures are the same as mentioned above, not #6793 (comment). That's a good thing--at least it's deterministic :)

With iTerm.app the cursor doesn't change shape for me. If I use a modified terminfo with Ss=\E[%p1%d q:Se=\E[2 q then the cursor changes shape, but I can't switch it to the vertical bar, only block or horizontal bar. Anyone else having the same issue?

Funny that you mention tests. (-:

• The test suite is now somewhat bigger. The Interix test failure seems to be a failure in the test harness, which rather assumes that any given terminal type is something that the :terminal terminal emulator can itself cope with. The Interix terminal type is rather different.
• Nicolas Marriott, Egmont Koblinger, the people who added true-colour support to emacs, and others all seem to agree on the idea of a new terminfo entry; so that is what nvim now employs. I intentionally did not invent nvim's own name for the capabilities, but went with one of the existing names. :help true-color has the details.
• As threatened, the Big Blocks Of Numbers are now thinner, but deeper. Enjoy.

@choco, if you are explicitly including the Ss and Se terminfo capabilities in your terminfo database, then nvim is bypassing all of the code to set them up itself, and is just using them as-is. There is, or at least should be, no terminal type checking or dependency in that path, no variation according to terminal type or anything else. nvim just spits out the relevant sequences from terminfo in that case.

• What happens when you just printf '\x1b[5 q' and printf '\x1b[6 q'?
• What happens when you just tput Ss 5 and tput Ss 6?

The observed behaviour implies, to me, that your terminfo isn't actually modified as you think. Because in the case where terminfo does not have these capabilities, and genuine Xterm is not detected, nvim fakes capabilities that don't include the xterm extension for vertical bar. (It seems that iTerm2 is another supporter of the Xterm DECSUSR extensions. I'll add it in. But that is a code path that according to what you say, you aren't exercising.)

The Interix test failure turned out to be in part a missing capability in the Interix terminfo record.

@jdebp You're right, actually my terminfo was pretty messed up, fixed that all works perfectly, sorry for the trouble.

@jdebp Actually there are still a couple of issues on my machine:

1. with this simple vimrc
   hi! Normal ctermbg=1
   i noticed that if I start nvim nvim -u testrc and start writing a command in the command line then press backspace the ui starts to break
2. vim doesn't render italics text, even though my terminal supports it and terminfo reports it (echo `tput sitm`italics`tput ritm` correctly print text in italic

App: iTerm2.app
$TERM: xterm-256color
terminfo:

xterm-256color|xterm with 256 colors,
	am, bce, ccc, km, mc5i, mir, msgr, npc, xenl,
	colors#256, cols#80, it#8, lines#24, pairs#32767,
	acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
	bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l,
	clear=\E[H\E[2J, cnorm=\E[?12l\E[?25h, cr=^M,
	csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
	cud=\E[%p1%dB, cud1=^J, cuf=\E[%p1%dC, cuf1=\E[C,
	cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
	cvvis=\E[?12;25h, dch=\E[%p1%dP, dch1=\E[P, dim=\E[2m,
	dl=\E[%p1%dM, dl1=\E[M, ech=\E[%p1%dX, ed=\E[J, el=\E[K,
	el1=\E[1K, flash=\E[?5h$<100/>\E[?5l, home=\E[H,
	hpa=\E[%i%p1%dG, ht=^I, hts=\EH, ich=\E[%p1%d@,
	il=\E[%p1%dL, il1=\E[L, ind=^J, indn=\E[%p1%dS,
	initc=\E]4;%p1%d;rgb\:%p2%{255}%*%{1000}%/%2.2X/%p3%{255}%*%{1000}%/%2.2X/%p4%{255}%*%{1000}%/%2.2X\E\\,
	invis=\E[8m, is2=\E[!p\E[?3;4l\E[4l\E>, kDC=\E[3;2~,
	kEND=\E[1;2F, kHOM=\E[1;2H, kIC=\E[2;2~, kLFT=\E[1;2D,
	kNXT=\E[6;2~, kPRV=\E[5;2~, kRIT=\E[1;2C, kb2=\EOE, kbs=^H,
	kcbt=\E[Z, kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,
	kdch1=\E[3~, kend=\EOF, kent=\EOM, kf1=\EOP, kf10=\E[21~,
	kf11=\E[23~, kf12=\E[24~, kf13=\E[1;2P, kf14=\E[1;2Q,
	kf15=\E[1;2R, kf16=\E[1;2S, kf17=\E[15;2~, kf18=\E[17;2~,
	kf19=\E[18;2~, kf2=\EOQ, kf20=\E[19;2~, kf21=\E[20;2~,
	kf22=\E[21;2~, kf23=\E[23;2~, kf24=\E[24;2~,
	kf25=\E[1;5P, kf26=\E[1;5Q, kf27=\E[1;5R, kf28=\E[1;5S,
	kf29=\E[15;5~, kf3=\EOR, kf30=\E[17;5~, kf31=\E[18;5~,
	kf32=\E[19;5~, kf33=\E[20;5~, kf34=\E[21;5~,
	kf35=\E[23;5~, kf36=\E[24;5~, kf37=\E[1;6P, kf38=\E[1;6Q,
	kf39=\E[1;6R, kf4=\EOS, kf40=\E[1;6S, kf41=\E[15;6~,
	kf42=\E[17;6~, kf43=\E[18;6~, kf44=\E[19;6~,
	kf45=\E[20;6~, kf46=\E[21;6~, kf47=\E[23;6~,
	kf48=\E[24;6~, kf49=\E[1;3P, kf5=\E[15~, kf50=\E[1;3Q,
	kf51=\E[1;3R, kf52=\E[1;3S, kf53=\E[15;3~, kf54=\E[17;3~,
	kf55=\E[18;3~, kf56=\E[19;3~, kf57=\E[20;3~,
	kf58=\E[21;3~, kf59=\E[23;3~, kf6=\E[17~, kf60=\E[24;3~,
	kf61=\E[1;4P, kf62=\E[1;4Q, kf63=\E[1;4R, kf7=\E[18~,
	kf8=\E[19~, kf9=\E[20~, khome=\EOH, kich1=\E[2~,
	kind=\E[1;2B, kmous=\E[M, knp=\E[6~, kpp=\E[5~,
	kri=\E[1;2A, mc0=\E[i, mc4=\E[4i, mc5=\E[5i, meml=\El,
	memu=\Em, oc=\E]104\007, op=\E[39;49m, rc=\E8, rev=\E[7m,
	ri=\EM, rin=\E[%p1%dT, ritm=\E[23m, rmacs=\E(B,
	rmam=\E[?7l, rmcup=\E[?1049l, rmir=\E[4l, rmkx=\E[?1l\E>,
	rmm=\E[?1034l, rmso=\E[27m, rmul=\E[24m,
	rs1=\Ec\E]104\007, rs2=\E[!p\E[?3;4l\E[4l\E>, sc=\E7,
	setab=\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48;5;%p1%d%;m,
	setaf=\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38;5;%p1%d%;m,
	sgr=%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p5%t;2%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m,
	sgr0=\E(B\E[m, sitm=\E[3m, smacs=\E(0, smam=\E[?7h,
	smcup=\E[?1049h, smir=\E[4h, smkx=\E[?1h\E=,
	smm=\E[?1034h, smso=\E[7m, smul=\E[4m, tbc=\E[3g,
	u6=\E[%i%d;%dR, u7=\E[6n, u8=\E[?%[;0123456789]c,
	u9=\E[c, vpa=\E[%i%p1%dd,

3d7b674 will have already fixed your command-line problems, @choco. It was deferred right margin wrap throwing off the left-motion calculations.

Italics is mystifying. If it is in the terminfo entry, it should be pulled out and used as-is.

3d7b674 will have already fixed your command-line problems, @choco. It was deferred right margin wrap throwing off the left-motion calculations.

Actually just tried and still I'm still having that issue.

All of the remaining problems appear to be MacOS, and things that I cannot reproduce.

• The colour functionality test is failing when TERM=linux-16color on MacOS. It isn't failing on the number of colours. That, it gets right. It's failing by drawing the command line in the wrong place on the screen. But the exact same TERM=linux-16color test is succeeding on other platforms.
• @choco reports an italics failure in iTerm.app. I have no access to iTerm.app. But on every terminal where printf '\x1b[3mitalics\x1b[23m upright' gives me italics, so too does TERM=xterm-256color nvim -u NONE -c ":terminal printf '\\x1b[3mitalics\\x1b[23m upright'".
• @choco reports "the ui starts to break" in iTerm.app when using backspace on the command line. The best that I can do is make nvim think that the terminal is iTerm with TERM=iterm or TERM=xterm TERM_PROGRAM=iTerm.app and then run it through some other terminal emulator. But I cannot get it to exhibit any command-line problems with backspace that way.

@jdebp Mmmmm the italics issue was probably my fault, since modifying manually the macOS standard terminfo fixed that, but the second issue I'm sure can't be caused by me.
Tested on two different machines (one just reset) and nvim -u NONE -c ":hi! Normal ctermbg=1" once entered command line and entered some characters, pressing backspace corrupts the ui, both in Terminal.app and iTerm.app with both using $TERM=xterm-256color

One thing to try @choco:

TERM=xterm-256color is not actually correct for either of those terminal emulators. The xterm-256color terminfo entry does not describe them correctly.

• The correct terminal type for iTerm.app is TERM=iterm or TERM=iTerm.app. Yes, nvim will handle this terminal type even on MacOS. See :help builtin-terms.
• The correct terminal type for Terminal.app is TERM=nsterm.

Similarly, for anyone else, the correct terminal type for GNOME Terminal is TERM=gnome, TERM=vte, TERM=gnome-256color, or TERM=vte-256color and Thomas E. Dickey has been complaining about GNOME Terminal setting the variable wrongly since at least 2006, given that terminfo has had the gnome terminal type since 1999.

If you had to modify the vanilla MacOS terminfo, @choco, then nvim really needs to as well. Presumably you took the iTerm.app|iterm entry and added the missing sitm. If not, what did you do?

@jdebp I'm using xterm-256color because that's what both terminal app use as default.
On my system there's no terminfo named iterm or iTerm.app and the nsterm terminfo is incredibly old, it refers to Terminal.app v41, when current is 388!

I tried updating ncurses thought homebrew, the package manager I use on macOS, and installed v6.0 and force linked to use the included terminfo. When doing so, using nsterm on Terminal.app now uses the same terminfo on the page you linked (nsterm-build361) but I can still reproduce the bug above (the command line one).

The same goes for iTerm2.app, the terminfo above isn't installed by default, only when doing the procedure above; even after all that, it still has the same issue. I also wanted to note that the terminfo in that page refers to iTerm version 1, but the current one is v2 (actually version 3, but it's still called iTerm2) which does probably support many more features. As far as I know iTerm2 exposes a proprietary control sequence to check for available terminal features, since terminfo databases are really outdated on macOS and that's what tmux uses to query supported features I think.
We should probably tag @gnachman (iterm developer), if he has some time to spare, who can maybe explain the situation better then I can.

Looks like the functional tests are passing, but CI builds are failing by -Werror, numerous warnings like this:

tui.c:2421:58:� �error: �negative integer implicitly converted to unsigned type [-Werror=sign-conversion]
-1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,  -1,

Problem in Guake

@SanD94 It fixes your problem?

@Shougo I couldn't fix that.

guake version : 0.8.4-1

I find that printf '\x1b[q 2' or printf '\e[2 q' in Guake does not give me the result I get when using gnome-terminal, block cursor without blink.
However, there is an option to change cursor shape in guake and all cursor shape (block, I-beam and underline) seems working.

ps : sorry, I am not familiar with these stuff and I don't know the meanings of \x1b[q 2 or \e[2 q

ps : sorry, I am not familiar with these stuff and I don't know the meanings of \x1b[q 2 or \e[2 q

It is the cursor shape escape code.

However, there is an option to change cursor shape in guake and all cursor shape (block, I-beam and underline) seems working.

It just change the cursor shape by user option.
It is not same with changing the cursor shape by the escape code.

The issues this PR fixes basically make nvim at HEAD hard to use with multiple splits (which is common). The thread is getting quite long, can anyone make a list of the points this PR regresses on in comparison to current HEAD? If there are none, I'd suggest to merge this and do the rest in separate PRs. Anyway, thanks for all the work already, @jdebp.

The change to screen:expect is still wrong. Can use the suggested fix or just drop the change (I don't see any immediate-success being added, so thus unrelated to this PR), I hope to improve expect robustness in general (with reasonable timeouts) in #6930 anyway.

Planning to merge this without the changes to test infrastructure (prefer #6930 for that).

👍 for merging this soon, since I know that people are not using the nightly anymore because of that.

Just compiled this pull request and it fixed the issue for me, which is a ^_^
Actually, I merged master into this PR before compiling. No conflicts whatsoever.

Thanks!

Yeah, it seems to be mostly via #6802 (comment) that people are affected - IIRC things were merged too quickly, and fixes for it (this PR) is taking too long.

Merged after subtracting the test-suite changes (e2fba01 7c809c4 18e7cd9). Thanks to @jdebp for this wonderful improvement.

After this getting merged, 'termguicolors' no longer works properly with tmux 2.5 (using $TERM=tmux-256color). I'm getting greyscale colors instead of real colors.

I can't tell from the new documentation if there's something I'm supposed to do either to my tmux config or terminfo to fix this.

FWIW, infocmp tmux-256color doesn't report a value for initc, if that's relevant. I also tried adding the Tc terminal override to my tmux config, but either I'm doing it wrong or it doesn't have an effect.

@jamessan same thing but with alacritty instead of tmux

see #6982