Problem

When two diagnostics of different severity are emitted and one is contained within the other, the gui underline styles (undercurl, underdouble, underdash, underdot, underline) don't render as expected for the inner span. For example here are a few examples with an error contained inside a warning:

• dot containing dash: renders correctly.
• dash containing dot the inner error is dashed rather than dotted
• curl containing line: the inner error is double instead of single underlined???
• curl containing dash: the inner underline has completely disappeared 😧.

Steps to reproduce

I've written a minimal example using the diagnostics api: nvim --clean repro.lua -c 'so %' where repro.lua contains:

-- this line contains overlapping diagnostics

local ns = vim.api.nvim_create_namespace "test_namespace"
vim.diagnostic.set(ns, 0, {
    {
        lnum = 0, col = 10, end_col = 40,
        severity = vim.diagnostic.severity.ERROR, message = "error",
    }, {
        lnum = 0, col = 15, end_col = 30,
        severity = vim.diagnostic.severity.WARN, message = "warning",
    },
})

vim.cmd [[
hi DiagnosticUnderlineError guisp='Red' gui=underline
hi DiagnosticUnderlineWarn guisp='Cyan' gui=undercurl
set termguicolors
]]

The settings in ^ produce "line containing curl" underline, which is one of the combinations that actually renders correctly for me. I can switch out the error and warn gui settings and :so % to see how the different ones behave.

Also this probably warrants a separate issue but the :h docs use "underdotted", "underdashed" etc. but those don't seem to actually work. you have to use "underdot", "underdash", etc.

The real-world usage where I ran into this was with LSP diagnostics. rust-analyzer emits some overlapping diagnostics and I had different underline styles that happened to be some of the combinations that render incorrectly. The easiest example to reproduce I've found is adding use asdf; to any rust file with rust-analyzer enabled. You get a warning diagnostic about the unnecessary single-path-component import, and also an error diagnostic with a span completely contained within the former about the unknown crate asdf. If the diagnostic-underline-warn/error highlight groups have different styles then the issue occurs.

Expected behavior

I'd expect arbitrary sequences of underline styles to display correctly. For example outside of neovim the following renders correctly (using kitty):

echo -e '\e[4mline\e[4:2mdouble\e[4:3mcurl\e[4:4mdot\e[4:5mdash'

Within neovim I've manually set some non-diagnostics highlights and they also render correctly when using combinations that the diagnostics mishandled. nvim --clean repro.vim -c 'so %' where repro.vim contains:

hi vimEscape gui=undercurl
hi vimString gui=underdash
set termguicolors

let x = "as\n\n\ndf" " note the correct mixing

I think the difference here is that those groups aren't actually overlapping, it just switches back from one to the other. I don't know of any cases other than diagnostics underlines where you could have highlight groups that overlap and combine, but if those exist I would guess they have the same problem.

Neovim version (nvim -v)

v0.9.0-dev-901+g563881306

Vim (not Nvim) behaves the same?

Haven't checked, but I don't think vim handles diagnostics the same way as neovim so i'm not sure how i'd reproduce

Operating system/version

Debian 5.19.11

Terminal name/version

kitty 0.27.1

$TERM environment variable

xterm-kitty

Installation

build from repo