Lexers is a Julia library for easy implementation of lexers (aka tokenizers).

It has no dependencies and has support for Julia 1.6 and later.

One of the interesting selling points of Lexers is that the different Token and Lexer kinds are implemented into the type system.

For example, a minimal lexer for the Brainfuck language can be implemented as

using Lexers

Brainfuck = Lexer{Tuple{GreaterThan, LessThan, Plus, Minus, Dot, Comma, BracketLeft, BracketRight}}

where GreaterThan, LessThan, Plus, ... are token types.

In order to lex or tokenize a code string, call the tokenize function.

NOTE: This code result is not yet loyal to reality.

julia> tokenize(Brainfuck, "[->+<]")
6-element Vector{Lexeme{T, SubString{String}} where T<:Token}:
 "[" => Bracket left
 "-" => Minus
 ">" => Greater than
 "+" => Plus
 "<" => Less than
 "]" => Bracket right