alasdairforsythe / tokenmonster Star 539 Code Issues Pull requests Discussions Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript tokenizer vocabulary vocabulary-builder tokenize tokenization tokenisation tokenizing text-tokenization vocabulary-generator Updated Jul 2, 2024 Go
bzick / tokenizer Star 89 Code Issues Pull requests Tokenizer (lexer) for golang golang parser parse tokenizer lexer tokenizing Updated Apr 9, 2024 Go