feat(lsp): add support for semantic tokens

[smolck]

Relevant part of the spec: https://microsoft.github.io/language-server-protocol/specifications/specification-current/#textDocument_semanticTokens

[mjlbach]

You need to add the capability here where we resolve/sanitize server capabilities (do a real check, this was just a dirty fix)

diff --git a/runtime/lua/vim/lsp/protocol.lua b/runtime/lua/vim/lsp/protocol.lua
index 6e98ea1f4..27cbde164 100644
--- a/runtime/lua/vim/lsp/protocol.lua
+++ b/runtime/lua/vim/lsp/protocol.lua
@@ -1016,6 +1016,7 @@ function protocol.resolve_capabilities(server_capabilities)
   general_properties.document_range_formatting = server_capabilities.documentRangeFormattingProvider or false
   general_properties.call_hierarchy = server_capabilities.callHierarchyProvider or false
   general_properties.execute_command = server_capabilities.executeCommandProvider ~= nil
+  general_properties.semantic_tokens_full = true
 
   if server_capabilities.renameProvider == nil then
     general_properties.rename = false

Adding this, seems like you're missing textDocument as part of the params you're sending:

[ DEBUG ] 2021-03-14T23:57:58-0700 ] /usr/local/share/nvim/runtime/lua/vim/lsp/rpc.lua:391 ]	"rpc.send.payload"	{  id = 2,  jsonrpc = "2.0",  method = "textDocument/semanticTokens/full",  params = {    uri = "file:https:///Users/michael/Repositories/rust/strings/src/main.rs"  }}
[ DEBUG ] 2021-03-14T23:57:58-0700 ] /usr/local/share/nvim/runtime/lua/vim/lsp/rpc.lua:492 ]	"decoded"	{  error = {    code = -32602,    message = 'Failed to deserialize textDocument/semanticTokens/full: .: missing field `textDocument`; {"uri":"file:https:///Users/michael/Repositories/rust/strings/src/main.rs"}'  },  id = 2,  jsonrpc = "2.0"}

You can use vim.notify if your prints are being swallowed, I have this handy snippet in my config:

local scratch_buffer = vim.api.nvim_create_buf(false, false)

local function notify(msg, log_level, _opts)
  if not vim.api.nvim_buf_is_valid then
    scratch_buffer = vim.api.nvim_create_buf(true, false)
  end
  vim.schedule(function() vim.api.nvim_buf_set_lines(scratch_buffer, 0, 0, false, vim.split(msg, '\n')) end)
end
vim.notify = notify

local opened_scratch = false
local win_id
function OpenScratch()
  if vim.api.nvim_win_is_valid(win_id) and opened_scratch then
    vim.api.nvim_win_close(win_id)
    opened_scratch = false
  else
    win_id = vim.api.nvim_open_win(scratch_buffer, false,  {relative='editor', row=10, col=140, width=60, height=120})
    opened_scratch = true
  end
end
vim.api.nvim_set_keymap('n', '<space>so', ":lua OpenScratch()<CR>", { noremap=true })

[Congee]

Ping

Is there any update for this exciting feature?

[smolck]

@Congee I'm happy to continue working on this, but I wasn't quite sure exactly where we wanted to go with it, specifically how to use the tokens by default. My initial idea was to have semantic highlighting, but that feels more like something for an external plugin to deal with (analogous to nvim-treesitter using the builtin tree-sitter things to add highlighting etc.) and isn't as easy to deal with as it would initially seem, and so I'm a little lost at this point.

Maybe a good default functionality would be to store the tokens in a cache and handle creating/cleaning up that for buffers etc.? Then a plugin could retrieve them for use in highlighting or otherwise . . . overall though I'm not sure.

[Congee]

@smolck thank you for the update. Yeah maybe the semantic highlighting feature could be solely implemented by a plugin like https://github.com/jackguo380/vim-lsp-cxx-highlight when nvim-treesitter still is not robust for now. Yet after searching through the some neovim communities, this PR is the only relevant work i found that the builtin neovim lsp can make use of.

Thank you for your pioneer work anyway.

[shadmansaleh]

@smolck I wanted to do this kind of things entirely from external plugin with #14322 but you and @mjlbach were against it :) . I think treesitter highlighter is in core so is regex highlighter . It might not be too bad to do semantic highlighting directly from core .

[smolck]

I think treesitter highlighter is in core so is regex highlighter . It might not be too bad to do semantic highlighting directly from core .

Hmm that's a good point; I guess nvim-treesitter just takes care of parser install and such, but the main functionality is in core, didn't think about that. In any case, after a quick discussion w/mjlbach on gitter I think I know where to go with this now, should be able to move things along this evening if all goes well.

TL;DR is we'll have a way to retrieve the tokens via a function given a client & bufnr, for plugins to do with as they wish. That would enable me to make a plugin for semantic highlighting to flesh that out, and then maybe get it merged into core eventually if we decide we want to have semantic highlighting in core.

[shadmansaleh]

I think I know where to go with this now, should be able to move things along this evening if all goes well.

Cool 👍

TL;DR is we'll have a way to retrieve the tokens via a function given a client & bufnr, for plugins to do with as they wish. That would enable me to make a plugin for semantic highlighting to flesh that out, and then maybe get it merged into core eventually if we decide we want to have semantic highlighting in core.

Just saw the conversation on gitter . I agree this pr can just add this and later prs can add highlighter if needed .

TBH I don't understand why people are looking for this . TS already provides language aware highlighting 🤷‍♂️

[Congee]

TBH I don't understand why people are looking for this . TS already provides language aware highlighting man_shrugging

@shadmansaleh nvim-treesitter still is not robust enough that I have to frequently do :e. Some languages like Haskell or Scalaw does not have a query file for treesitter, either. 😢

[shadmansaleh]

vim-treesitter still is not robust enough that I have to frequently do :e.

true . But this is also very new .

Some languages like Haskell or Scalaw does not have a query file for treesitter, either. 😢

I feel like in genaral it might be easier to find a TS parser then a lsp server that supports semantic highlighting . Since building a TS parser takes a lot less effort then a lsp server. there seems to a haskell parser but I think it's unmaintained . seeing this I think semantic tokens aren't yet supported by haskell language server . the situation seems same for scala there's an unmaintained TS parser and the lsp server doesn't support semantic highlighting . Though I'm not 100% sure because I don't use these languages :)

[disrupted]

there seems to a haskell parser but I think it's unmaintained .

what makes you think it's unmaintained? No offense, I just had a brief look and it looks pretty active to me, last commit is under 1 month ago.

[smolck]

TBH I don't understand why people are looking for this . TS already provides language aware highlighting 🤷‍♂️

I think I have a bit of an understanding of why, but rather than try to explain it myself (which I'm not sure I can do well), I'll just link to this issue on nvim-treesitter and specifically this comment on that issue by the author of semshi.

From looking at the end of that issue, it seems tree-sitter does support semantic highlighting, at least in some form. However, I'm sure there are valid usecases for LSP-based semantic highlighting, and perhaps at this point and/or in some cases it is better than tree-sitter highlighting, I don't know.

Regardless, none of this means we shouldn't expose semantic tokens to users/plugin authors to do with as they please. The question of whether or not semantic highlighting should be included in core is not something this PR addresses; instead the plan is to only make such a feature (and any others using semantic tokens) possible, whether in a plugin or in core itself, by exposing semantic tokens to users/plugin authors.

[shadmansaleh]

what makes you think it's unmaintained?

Sorry didn't try to offend anyone
I was looking at the list on nvim-treesitter repo and haskell and scala didn't have check mark and didn't have a maintainer name that's why I thought it was unmaintained .

, I'll just link to this issue on nvim-treesitter and specifically this comment on that issue by the author of semshi.

Thanks for the links I'll read them :)

[Congee]

Yeah, There's still additional work needed to have Semshi like highlighting.

@disrupted the Haskell treesitter integration had been unmaintained over 2+ years until recently, the README of nvim-treesitter hasn't caught up yet 😅

@theHamsta just tried the latest revision tree-sitter/tree-sitter-haskell@2e33ffa, It works for a few lines of code with the only feature incremental selection. And TSModule says highlight and indent are not supported yet.

I am happy to see the community getting active on all these issues 😁

[clason]

Two things (that were mentioned before but only in passing):

1. If tree-sitter based highlighting -- which has been and still is in very active development -- isn't robust yet, what makes you think that semantic highlighting -- which is still at the WIP stage -- is going to be a replacement for production use any time soon! (Not that I don't have full confidence in @smolck, but I sense unreasonable expectations building up again and want to make sure they are managed).
2. Semantic highlighting in general doesn't make sense as a replacement for regex or treesitter highlighting; LSP is challenging to make responsive enough for it, especially on large projects. (People already complained about tree-sitter not being fast enough and showing flicker!) On the other hand, LSP can do things that tree-sitter can't (easily), namely highlight based on information from other files -- for example, if a variable is defined as const in foo.h and you use it in bar.c, it can get a different highlight in the latter to distinguish it from non-const variables. So I think it is important to design semantic highlighting support from the ground up to be a supplement to rather than a replacement of whatever "base" highlighting (regex or tree-sitter) is used.

[theHamsta]

since the module is already called semantic tokes I guess you can drop it to make the function names more terse

[smolck]

@theHamsta So like _handle_full()? Or what did you have in mind exactly?

[theHamsta]

I would prefer it. Especially for 'lsp.semantic_tokens'.get_semantic_tokens() a 'lsp.semantic_tokens'.get() would feel better. But I get is feels also a bit unsafe for external users since they must know when to request tokens. Maybe a subscription to token changes would be better?

I also played a bit around with this PR: https://github.com/theHamsta/neovim/blob/81f20e363c95437b12eec600eb003a9b54b5d7d4/runtime/lua/vim/lsp/semantic_tokens.lua#L1

[williamboman-pp]

vim.lsp.buf_get_clients({bufnr})

[theHamsta]

on_refresh is triggered by a particular language server so we should also use this server that requested a refresh: https://github.com/theHamsta/neovim/blob/33de6be20b21a250a9374d25bd0fb9e6cb288664/runtime/lua/vim/lsp/semantic_tokens.lua#L113 (but actually using the client_id 😅 )

[theHamsta]

why not use the same names as from the LSP spec?

    local token = {
      line = line,
      start_char = start_char,
      length = data[i + 2],
      type = token_type,
      modifiers = modifiers
    }

[clason]

(yes, please -- the closer we stick to the spec, the less confusion)

[theHamsta]

Just line? The line is not really previous but also current

[theHamsta]

if and else branch are essentially the same code. Maybe?

    local token = {
      line = line,
      start_char = start_char,
      length = data[i + 2],
      type = token_type,
      modifiers = modifiers
    }
    tokens[line + 1] = tokens[line + 1] or {}
    table.insert(tokens[line + 1], token)

[neovim-discourse]

This pull request has been mentioned on Neovim Discourse. There might be relevant details there:

https://neovim.discourse.group/t/lsp-and-semantic-token-support/784/2

[jackhub]

any update here?

[smolck]

@jackhub Haven’t touched it in a while for a few reasons, one of which being I’ve been waiting on a review from @mjlbach. I do need to get around to @theHamsta’s reviews though, which I should be able to do tomorrow.

[tom-anders]

@smolck @theHamsta Do you need some help with testing? I could test it with clangd, if you haven't already.

[tom-anders]

Any updates here?

[justinmk]

continued at #15723