Reduce parser size and compile time

[maxbrunsfeld]

This PR makes a few optimizations to the grammar, to reduce the size of the generated parser, and also regenerates the parser with the latest tree-sitter CLI, which contains tree-sitter/tree-sitter#3234, an optimization to the compile time of the generated C code, which affects this parser significantly.

This should unlock compiling this parser to WASM and packaging it in a Zed extension. Compiling to wasm is still slow (about 45 seconds on my M3 MacBook). Compiling to aarch64 only takes 1.5 seconds. I still don't understand why clang is so much slower when compiling to wasm than when compiling to native.

I believe that the reason that this parser compiles so much slower than many others is that its generated ts_lex function has a large number of states, which I think is due to some complex tokens in this grammar:

• unquoted and unquoted_in_list, which both expand to fairly complex sequence of different anonymous tokens defined by separate regexes.
• the three different identifier rules.

I feel like there is probably a lot of room to simplify some of this, but I didn't attempt it in this PR.

/cc @fdncred

[fdncred]

Thanks @maxbrunsfeld! We appreciate your support. Wow, 60 files changed. I didn't expect that. Please excuse my ignorance but what are all these extra files used for? There seem to be a lot of files we didn't have before.

[maxbrunsfeld]

A lot of the changes files are just moving the corpus test folder under ‘test’, because that’s where the new tree-sitter CLI looks for them. There are also some binding files generated, to make the parser usable from Go, Python, and Swift I think.

[fdncred]

Can you fix the prettier lints with npx prettier grammar.js --write?

[maxbrunsfeld]

Yes. I will; I'm also going to regenerate the parser one more time; I found a small improvement I'd like to make on the Tree-sitter side.

[fdncred]

Thanks. Let's move forward with this. If we decide we don't like the extra files (makefile, editor config, extra bindings), we can always remove them later. It's just software. :)

[maxbrunsfeld]

Yeah, tree-sitter generates those now, in case Python/go users want to use the parser. I agree, it is a lot of files.

I kind of agree that the editor config seems unnecessary. I might remove that file from the generate code.

[fdncred]

We really appreciate your help. ❤️

I also created a new PR to the zed-extension/nu repo in order to update to this version.