Library functions are not identified by Scope selectors

[necroze]

Hello,
in the syntax definition of powerquery - scope selector for library functions is absent.
Parts of a function(List.Generate for example)
"List" part shows "entity.name" scope
".Generate" part shows "no scope selector"

There are numerous ways to fix that - from hardcoding function names ending to adding a proper regex into syntax file.

Please do something. It is important for me and lots of other PBI developers(I asked =))

[JordanBoltonMN]

I'm not sure what you are exactly referring to by scope selector. It sounds like you're referring to namespaces, eg. that Text should hold all text functions?

[necroze]

I mean that both sides of the function name should respond to same syntax scope e.g. "entity.library-function.powerquery". Now only the left side of "." is identified by syntax and right side has none at all.

[necroze]

No actions planned on the issue?

[JordanBoltonMN]

I'm trying to understand the exact issue being raised. Is it that the powerquery.tmLanguage.json is incorrect in that it's splitting tokens on the periods, eg. Left.Right?

[necroze]

Absolutely! And there can be 2 types of mistakes either it's pattern/capture structure or regex of the definition. I suspect first though.

[necroze]

(([A-Z][A-Za-z0-9_]+)(?=(.)).(\w*\b))

For example a simple regex like this one will catch both sides of Left.Right statements and nothing more...

[necroze]

Oh. C'mon aren't there someone who can adjust a tmLanguage.json file so all the scopes are included properly?

[JordanBoltonMN]

@necroze I've been pulled in a few directions of late where I haven't had time to verify that your suggestion works as expected. If you are interested the repository is open source and you can submit a PR which I'll try to make time to verify.

[necroze]

@JordanBoltonMN I am really interested in repo but right now I got an all out war in my country. Thus I am distracted from the matter. I hope we will be able to get back to it in the future, when it’s safe again.

[ninmonkey]

I hope you're safe. I felt like replying since there hasn't been an update.

The short answer: It's not likely to go into main because there's no classes in Power Query. Table.RemoveColumns is a single identifier -- with a period in it. It's misleading and looks strange, because that's not a valid literal in many languages. It would break things or be a member operator.

`Table.RemoveColumns` is a regular single identifier/token, just like `x` It's basically the same thing as these
let sort = List.Sort in sort( {3, 1, 10} )	let Foo.Bar.Cat = each + _ 1 in Foo.Bar.Cat( 10 )

You can see exactly what characters (codepoints) are valid, if you go down the grammar tree

Some Textmate Customization in your profile

For your screenshot did you modify your settings.json or the extension's syntaxes/powerquery.tmLanguage.json ? Reading the docs, I couldn't tell if I can actually add a regex -- without editing an addon. The closes I found was
In your settings.json

"editor.tokenColorCustomizations": {
        "textMateRules": [
            {
                "scope": "comment.line", // single-line comments
                "settings": {
                    "foreground": "#6ea58049",
                    "fontStyle": "bold"
                } } ] }

Local override

The easiest method might be to edit your local grammar file. To find the latest tmLanguage file, you can use this command in powershell:

Pwsh> (ls "$Env:USERPROFILE/.vscode/extensions/*powerquery*/syntaxes/powerquery.tmLanguage.json" | sort LastWriteTime)[-1].FullName

Mine is $Env:UserProfile/.vscode/extensions/powerquery.vscode-powerquery-0.1.40/syntaxes/powerquery.tmLanguage.json

Stress Testing + Semantic highlight

I was curious so I made a stress test, your regex works surprisingly well. Note: I switched the dialect to dotnet while experimenting, so I could use [RegexOptions]::IgnorePatternWhitespace to make it readable. It should be fairly close to the Javascript one.
https://regex101.com/r/t0uUpJ/3

There's some sort of fallbacks to TM when using semantic highlighting, if that were enabled, maybe you can force certain tokens to fallback to textmate, to colorize it like a class. Then you could hook into one specific token type -- making your regex simplerm and more accurate