Parser Building Toolkit for JavaScript
-
Updated
Aug 22, 2024 - TypeScript
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Parser Building Toolkit for JavaScript
VSCode extension to highlight nested code blocks
Contains source-code for viewers following along with my Beginners Guide To Building Interpreters series on my Youtube Channel.
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
An extension for the Tiptap editor, enabling the annotation of text. Comes with support for overlapping annotations, useful for e.g. NLP tokenization.
🐉 JavaScript expression tokenizer and evaluator
📐 GPT token estimation and context size utilities without a full tokenizer
A primitive token library used to read from, and to write a node Buffer
A simple lexer written entirely in JavaScript, with no dependencies!
n8n node for working with BPE Tokens with GPT in mind.
Play with OpenAI API's using your own API Key. Your API Key is stored and used only from your browser.
Text input augmented with Lexer magic✨ to support advanced search features.