feat(lsp): add support for semantic tokens

[theHamsta]

Another take on #14122 . I updated @smolck PR to the new LSP handler signatures and added some basic highlighting.

• reword commits to confirm conventional commits
• Align highlight mapping with the strategy of vim.treesitter.highlight (e.g. default link of thinks like cppLspComment)
  • current solution tries to have similar behavior as tree-sitter
  • mapping for token types: LspComment, cppLspComment automatically linked to LspComment
  • mapping for token modifiers: LspDeprecated, cppLspDeprecated
  • TODO: do we need a combination of type and modifier like LspFunctionDeprecated? or should be just provide a overwritable function that does the mapping ft x token x modifiers -> hl
• handle refreshing -> proper implementation in follow-up with partial refreshing, for now just full request on autocmd)
• implement partial refreshes of semantic tokens -> follow-up PR
• documentation

Fixes nvim-treesitter/nvim-treesitter#1632 @cryptomilk I think this would fit your use case. When your run autocmd BufEnter,CursorHold,InsertLeave <buffer> lua vim.lsp.buf.semantic_tokens_full() with clangd and LspComment linked to Comment.

semantic_tokens.mp4

Open question should the default semantic token handler handle highlighting? Should this be opt in?

• my opinion would be to handle the highlighting. Semantic token would still do no highlighting as the highlight groups are not mapped. Users would opt-in partially or completely by a example config that the docs provide

[mfussenegger]

For me the two issues that should to be addressed before this would be good to merge are:

a) A concept to handle multiple LSP clients. Either the API needs to support client filters (like vim.lsp.buf.format, or rename) or there would need to be some kind of aggregate logic. I'm not sure which variant makes more sense for semantic tokens. That's also missing from other parts like codelens, but we learned that retrofitting this can be a bit painful (e.g. due to breaking changes). It's better to consider this from the start then trying to shoehorn it on top later.

b) Showcase how multiple plugins would consume the tokens. (and probably improve the API for that, because I think each pluggin wrapping a function isn't ideal - it should probably be possible for one consumer to stop consuming / de-register itself - e.g. autocmds could be a solution as already mentioned)

Given that both of these are influencing the public API I'd rather see this addressed before merging, not after. But if another core maintainer disagrees and wants to push this forward I won't object.

[theHamsta]

#19931 could also be a possibility for mapping semantic token highlights in future #tokenType.modifierA.modifier or even the same format that VSCode uses

{
  "name": "Red Theme",
  "tokenColors": [
    {
      "scope": "comment",
      "settings": {
        "foreground": "#dd0000",
        "fontStyle": "italic"
      }
    }
  ],
  "semanticHighlighting": true,
  "semanticTokenColors": {
    "variable.readonly:java": "#ff0011"
  }
}

variable.readonly:java is called a selector and has the form (*|tokenType)(.tokenModifier)*(:tokenLanguage)?.

https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#semantic-coloring-in-color-themes

[tiagovla]

LSP 3.17 added decorator to the SemanticTokenTypes.

[christianparpart]

I was trying out branch, and I am running into some issues, sadly.

Nothing seems to be highlighted by default. So I scrolled through this PR and found out that you'd need to call the given function as follows to refresh highlighting. - But calling lua vim.lsp.buf.semantic_tokens_full() will result into an error, as follows:

Error executing vim.schedule lua callback: ...local/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:70: attempt to index local 'config' (a nil value)
stack traceback:
        ...local/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:70: in function 'handler'
        /usr/local/share/nvim/runtime/lua/vim/lsp.lua:1372: in function ''
        vim/_editor.lua: in function <vim/_editor.lua:0>

Also, terminating vim takes about 2 seconds on a really strong desktop machine, and I wonder what might hold it back. Some timeout maybe?

Regardless of the slow program-exit, I'd really like to give this branch a try because I need a way to test semantic tokens from the server side and would prefer to stick with (neo)vim for that. What am I maybe missing?

p.s.: Would you mind doing a git rebase ... on top of latest master? Seems like your base is pretty outdated already. :)

[lewis6991]

This PR doesn't implement buffer highlighting, it simply adds support for semantic tokens in the LSP client. You will need to implement the buffer highlighting yourself by providing a callback for on_token.

Also, terminating vim takes about 2 seconds on a really strong desktop machine, and I wonder what might hold it back. Some timeout maybe?

My guess is that you are running a debug build which defines -DEXITFREE. This will cause the hang on exit (see #18670). To avoid this simply compile a release build with make CMAKE_BUILD_TYPE=Release.

[iago-lito]

You will need to implement the buffer highlighting yourself by providing a callback for on_token

@lewis6991 Is it possible yet to highlight individual/contextualized tokens from this callback? Or would it essentially be still a sequence of :syn calls?

[lewis6991]

You wouldn't use :syn calls, you would use nvim_buf_set_extmark

[kaiuri]

@christianparpart

Regardless of the slow program-exit, I'd really like to give this branch a try because I need a way to test semantic tokens from the server side and would prefer to stick with (neo)vim for that. What am I maybe missing?

I suggest checking out nvim-semantic-tokens. Been using it daily without issues.

[jdrouhard]

It seems this PR has stalled out again :(

@theHamsta - it looks like you gave up a bit ago on actually pushing this through. Totally understandable, maybe I could open a continuation PR to keep going.

@clason @lewis6991 @mfussenegger - To address @mfussenegger 's #15723 (comment), I think we need to hone in on exactly what form semantic tokens takes. Your (a) bullet contains two possibilities - we should decide on one. (b) is already sorta done-- @theHamsta and I both have different plugins that can consume the tokens. If we think the API for that should be improved, we should hammer that out. If this is better suited to a matrix discussion in neovim-dev, I'd be happy to hop on and discuss at some point.

At this point, I think it's time we close this PR and open a new one that attempts to address whatever remaining things you guys want to see before we can get this in.

Semantic tokens is a pretty major feature of LSP that the native client still doesn't support and it would be great if we could get some support for it (even if rudimentary) in before 0.9.0.

[christianparpart]

@jdrouhard I absolutely feel with you. Last time I tried native LSP, the lag of semantic tokens was THE reason for me to switch back to coc.nvim. Since then I didn't even care about moving forward again, as no semantic highlighting is in fact a deal breaker for me. I can't complain too much here, as I can't contribute to it either. Lua isn't just my thing, so I'm happy for anyone to just finish this work such that others (not just me) could eventually start using the native LSP (again).

Meanwhile I keep watching, all the best,
Christian.

p.s.: one important note to drop. I was actively working on an LSP server myself, and the thing I've learned from it, is that users ARE actualy willing to take non-finished products gladly now rather than waiting a year or more until it is 100% complete. There's no such thing as perfect anyways. Release often and early they say. Use that as an advantage to re-iterate often, get early feedback, and improve while you go. ... and again, all the best to you guys. Cheres ;)

[mfussenegger]

. (b) is already sorta done-- @theHamsta and I both have different plugins that can consume the tokens

I haven't seen yours, but afaik the one from theHamsta uses something like vim.lsp.with(semantic_tokens.on_full, {on_token = highlight_token, on_invalidate_range = clear_highlights, }) - which means no two plugins can consume tokens at the same time. I don't see this as solved.

Since my last comment autocmds gained support for a data property that allows to send arbitrary data. I think this makes using autocmds a even more attractive option to emit the tokens.

If this is better suited to a matrix discussion in neovim-dev, I'd be happy to hop on and discuss at some point.

Yes, matrix may be the better place to hash out the API

---

Regarding other comments: Please refrain from adding more comments that add no more additional information. We have already concluded that semantic tokens would be good to have in core. What's missing is someone who does the work and resolves the remaining issues.

[theHamsta]

Superseeded by #21100