Using a class property to automatically populate tokens set

[fireundubh]

For token-heavy grammars, instead of manually defining a set of token names, you can populate that set using a class property.

class classproperty:
    def __init__(self, fget):
        self.fget = fget

    def __get__(self, owner_self, owner_cls):
        return self.fget(owner_cls)


class MyLexer(sly.Lexer)
    @classproperty
    def tokens(cls):
        return {x for x in cls.__dict__ if x.isupper()}

In the example above, the tokens class property returns a set of token names from the MyLexer object's uppercased field names.

This eliminates the need to manually add each token name as a string to a set as in:

class MyLexer(sly.Lexer)
    tokens = {
        'NE',
        'DIGITS'
    }

    NE = '!='

    @_(r'\b([0-9.]+)[f]{0,1}\b')
    def DIGITS(self, t):
        pass

It would be nice if the user could instruct sly.lexer._build to automatically collect token names from the lexer object though.