Additional features for choosing how to parse input

[impl]

Hi!

We implemented a kubectl-like JSONPath template language on top of Gval, which we have been using in our production environment for about a year now with great success. To implement it we needed to add additional features to the base library.

For the languages:

• Language.ChainEvaluable(ctx, parser): Immediately run this language when the next token is consumed by the given parser.
• PrefixLanguage(prefix, lang, next): When the parser encounters the given prefix, it switches to another language. When that language cannot consume any more tokens, control is returned to the calling language and the next callback is invoked. Higher level interface for Language.ChainEvaluable(...).
• Init(next): Rather than immediately calling parser.ParseExpression(), you can perform some setup work (like configuring the parser to handle whitespace or tokens a specific way).
• Alternate(next): Called if no other parsing construct matches the given input.
• Late(func(outer Language)) Language: Allow late-binding of languages. This way you can embed a language in itself recursively. This is useful for some template languages, or, say, Bash string/command interpolation.

For the token parser:

• A language stack (see above).
• SetMode, SetWhitespace, SetIsIdentRuneFunc: Modify the underlying Go text scanner with the corresponding configuration.

I also fixed a small bug in the infix operator merging strategy.

You can see our complete implementation of the JSONPath template language I mentioned here as an example:

• https://github.com/puppetlabs/relay-core/blob/9b88f85bbbfc2f7a0eff903abe06c91406dbb6ee/pkg/util/jsonpath/template.go
• https://github.com/puppetlabs/relay-core/blob/9b88f85bbbfc2f7a0eff903abe06c91406dbb6ee/pkg/util/jsonpath/template_test.go

Let me know if you have any questions or suggestions for improvements. I hope you find these additions as useful as we have!

[generikvault]

Hi Noah,

thank you for contributing such a great feature.

I saw just a few small issues. I will mark them directly in the code.

[generikvault]

And thanks for fixing some additional bugs.

[generikvault]

I'm pretty sure you don't need this and could use somethig like

var a = PrefixExtension('x', parseX)
var b = PrefixExtension('y', parseY)

func parseX func(context.Context, *Parser) (Evaluable, error){
  p.ParseSublanguage(b)
}

func parseY func(context.Context, *Parser) (Evaluable, error){
  p.ParseSublanguage(a)
}

[impl]

Hmm, indeed, both this function and PrefixLanguage are trivially built using the other additions, and with entirely public interfaces.

Removing PrefixLanguage seems harmless, but my only concern about removing Late is that users may not discover such a mechanism for implementing recursive invocation by themselves. In many parser combinator libraries, in addition to the base set of parser constructs, they also expose some "patterns" that are built on top of the primitives, like Sequence, Many, Optional, etc. Do you think it would it make sense to have a patterns.go that has these type of utility language constructors in it?

For now, I've removed it altogether, but let me know if you'd like me to add it back in a different spot.

[generikvault]

Yes that could be a problem. I think example tests may help.