PEG.js is a simple parser generator for JavaScript that produces fast parsers with excellent error reporting. You can use it to process complex data or computer languages and build transformers, interpreters, compilers and other tools easily.
PEG.js is still very much work in progress. There are no compatibility guarantees until version 1.0
This fork now develops independently as the initial library is in stagnation and I do not see absolutely any signs that in the next years it will be able to quit this period :(. In contrast work in this fork really is carried. Planned features to implement in the 2018:
- Ranges
- Import grammars
- Attributes
- Stream/incremental parser
- Drop useless features (optimize for code size)
- Raise coverage up to 90%
- Ability to extract comments of the grammar
- Features
- Getting Started
- Installation
- Generating a Parser
- Using the Parser
- Grammar Syntax and Semantics
- Error Messages
- Comments
- Compatibility
- Development
- Simple and expressive grammar syntax
- Integrates both lexical and syntactical analysis
- Parsers have excellent error reporting out of the box
- Based on parsing expression grammar formalism — more powerful than traditional LL(k) and LR(k) parsers
- Usable from your browser, from the command line, or via JavaScript API
Online version is the easiest way to generate a parser. Just enter your grammar, try parsing few inputs, and download generated parser code.
To use the latest features, fixes and changes of PEG.js, directly install from the repository:
$ npm install Mingun/pegjs#master
PEG.js generates parser from a grammar that describes expected input and can specify what the parser returns (using semantic actions on matched parts of the input). Generated parser itself is a JavaScript object with a simple API.
To generate a parser from your grammar, use the pegjs
command:
$ pegjs arithmetics.pegjs
This writes parser source code into a file with the same name as the grammar file but with “.js” extension. You can also specify the output file explicitly:
$ pegjs -o arithmetics-parser.js arithmetics.pegjs
If you omit both input and output file, standard input and output are used.
By default, the generated parser is in the Node.js module format. You can
override this using the --format
option.
You can tweak the generated parser with several options:
-a
,--allowed-start-rules
— comma-separated list of rules the parser will be allowed to start parsing from (default: the first rule in the grammar)--cache
— makes the parser cache results, avoiding exponential parsing time in pathological cases but making the parser slower-d
,--dependency
— makes the parser require a specified dependency (can be specified multiple times)-e
,--export-var
— name of a global variable into which the parser object is assigned to when no module loader is detected--extra-options
— additional options (in JSON format) to pass topeg.generate
-c
,--config
,--extra-options-file
— file with additional options (in JSON format) to pass topeg.generate
-f
,--format
— format of the generated parser:amd
,bare
,commonjs
,es
,globals
,umd
(default:commonjs
)-p
,--plugin
— makes PEG.js use a specified plugin (can be specified multiple times)--trace
— makes the parser trace its progress
NOTE: On the command line, unless it's a repeatable option, any option on the right side will take priority over either the same option mentioned before or it's counter part:
pegjs -f es -f bare
will setoptions.format
tobare
pegjs --no-trace --trace
will setoptions.trace
totrue
pegjs -a start,Rule -a Rule,Template
will setoptions.allowedStartRules
to[ "start", "Rule", "Template" ]
In Node.js, require the PEG.js parser generator module:
var peg = require("pegjs");
In browser, include the PEG.js library in your web page or application using the
<script>
tag. If PEG.js detects an AMD loader, it will define itself as a
module, otherwise the API will be available in the peg
global object.
To generate a parser, call the peg.generate
method and pass your grammar as a
parameter:
var parser = peg.generate("start = ('a' / 'b')+");
The method will return generated parser object or its source code as a string
(depending on the value of the output
option — see below). It will throw an
exception if the grammar is invalid. The exception will contain message
property with more details about the error.
You can tweak the generated parser by passing a second parameter with an options
object to peg.generate
. The following options are supported:
allowedStartRules
— rules the parser will be allowed to start parsing from (default: the first rule in the grammar)cache
— iftrue
, makes the parser cache results, avoiding exponential parsing time in pathological cases but making the parser slower (default:false
)dependencies
— parser dependencies, the value is an object which maps variables used to access the dependencies to module IDs used to load them;
valid only whenformat
is set to"amd"
,"commonjs"
,"es"
, or"umd"
(default:{}
)exportVar
— name of an optional global variable into which the parser object is assigned to when no module loader is detected; valid only whenformat
is set to"globals"
or"umd"
format
— format of the generated parser ("amd"
,"bare"
,"commonjs"
,"es"
,"globals"
, or"umd"
); valid only whenoutput
is set to"source"
(default:"bare"
)output
— if set to"parser"
(default), the method will return generated parser object; if set to"source"
, it will return parser source code as a string; if set to"ast"
, it will return grammar AST as object; array from any above-mentioned formats. In this case will return an object with the keys corresponding to values of an array and the values on keys corresponding to result of the specified outputplugins
— plugins to usetrace
— makes the parser trace its progress (default:false
)source
— this object will be passed to anylocation()
objects insource
property (default:undefined
). This object will be used even ifoptions.source
will be redefined in the grammar
Using the generated parser is simple — just call its parse
method and pass an
input string as a parameter. The method will return a parse result (the exact
value depends on the grammar used to generate the parser) or throw an exception
if the input is invalid. The exception will contain location
, expected
,
found
, and message
properties with more details about the error.
parser.parse("abba"); // returns ["a", "b", "b", "a"]
parser.parse("abcd"); // throws an exception
You can tweak parser behavior by passing a second parameter with an options
object to the parse
method. The following options are supported:
startRule
— name of the rule to start parsing fromtracer
— tracer to use
Parsers can also support their own custom options.
The grammar syntax is similar to JavaScript in that it is not line-oriented and
ignores whitespace between tokens. You can also use JavaScript-style comments
(// ...
and /* ... */
).
Let's look at example grammar that recognizes simple arithmetic expressions like
2*(3+4)
. A parser generated from this grammar computes their values.
start
= additive
additive
= left:multiplicative "+" right:additive { return left + right; }
/ multiplicative
multiplicative
= left:primary "*" right:multiplicative { return left * right; }
/ primary
primary
= integer
/ "(" additive:additive ")" { return additive; }
integer "integer"
= digits:[0-9]+ { return parseInt(digits.join(""), 10); }
On the top level, the grammar consists of rules (in our example, there are
five of them). Each rule has a name (e.g. integer
) that identifies the rule,
and a parsing expression (e.g. digits:[0-9]+ { return parseInt(digits.join(""), 10); }
) that defines a pattern to match against the
input text and possibly contains some JavaScript code that determines what
happens when the pattern matches successfully. A rule can also contain
human-readable name that is used in error messages (in our example, only the
integer
rule has a human-readable name) and any count of attributes.
Attribute — any metadata which can be attached to the rule or code block and be
used by a codegen or other passes. The parsing starts at the first rule,
which is also called the start rule.
A rule name must be a JavaScript identifier. It is followed by an equality sign (“=”) and a parsing expression. If the rule has a human-readable name, it is written as a JavaScript string between the name and separating equality sign. Rules need to be separated only by whitespace (their beginning is easily recognizable), but a semicolon (“;”) after the parsing expression is allowed.
The first rule can be preceded by an initializer — a piece of JavaScript code
in curly braces (“{” and “}”). This code is executed before the generated parser
starts parsing. All variables and functions defined in the initializer are
accessible in rule actions and semantic predicates. The code inside the
initializer can access options passed to the parser using the options
variable. Curly braces in the initializer code must be balanced.
Let's look at the example grammar from above using a simple initializer.
{
function makeInteger(o) {
return parseInt(o.join(""), 10);
}
}
start
= additive
additive
= left:multiplicative "+" right:additive { return left + right; }
/ multiplicative
multiplicative
= left:primary "*" right:multiplicative { return left * right; }
/ primary
primary
= integer
/ "(" additive:additive ")" { return additive; }
integer "integer"
= digits:[0-9]+ { return makeInteger(digits); }
The parsing expressions of the rules are used to match the input text to the grammar. There are various types of expressions — matching characters or character classes, indicating optional parts and repetition, etc. Expressions can also contain references to other rules. See detailed description below.
If an expression successfully matches a part of the text when running the generated parser, it produces a match result, which is a JavaScript value. For example:
- An expression matching a literal string produces a JavaScript string containing matched text.
- An expression matching repeated occurrence of some subexpression produces a JavaScript array with all the matches.
The match results propagate through the rules when the rule names are used in expressions, up to the start rule. The generated parser returns start rule's match result when parsing is successful.
One special case of parser expression is a parser action — a piece of JavaScript code inside curly braces (“{” and “}”) that takes match results of some of the the preceding expressions and returns a JavaScript value. This value is considered match result of the preceding expression (in other words, the parser action is a match result transformer).
In our arithmetics example, there are many parser actions. Consider the action
in expression digits:[0-9]+ { return parseInt(digits.join(""), 10); }
. It
takes the match result of the expression [0-9]+, which is an array of strings
containing digits, as its parameter. It joins the digits together to form a
number and converts it to a JavaScript number
object.
Appending i
right after either a literal or a a character set makes the match
case-insensitive. The rules shown in the following example all produce the same result:
a1 = "a" / "b" / "c" / "A" / "B" / "C"
a2 = "a"i / "b"i / "c"i
a3 = [a-cA-C]
a4 = [a-c]i
Unlike in regular expressions, there is no backtracking in PEG.js expressions.
For example, using the input "hi!":
// This will fail
HI = "hi" / "hi!"
// This will pass
HI = "hi!" / "hi"
// This will also pass
HI = w:"hi" !"!" { return w } / "hi!"
For more information on backtracking in PEG, checkout this excellent answer on Stack Overflow.
There are several types of parsing expressions, some of them containing subexpressions and thus forming a recursive structure:
- "literal"
- . (dot character)
- [characters]
- rule
- ( expression )
- expression *
- expression +
- expression |..|
- expression ?
- & expression
- ! expression
- & { predicate }
- ! { predicate }
- $ expression
- label : expression
- expression1 expression2 ... expressionN
- expression { action }
- expression1 / expression2 / ... / expressionN
Match exact literal string and return it. The string syntax is the same as in
JavaScript. Appending i
right after the literal makes the match
case-insensitive.
Match exactly one character and return it as a string.
Match one character from a set and return it as a string. The characters in the
list can be escaped in exactly the same way as in JavaScript string. The list of
characters can also contain ranges (e.g. [a-z]
means “all lowercase letters”).
Preceding the characters with ^
inverts the matched set (e.g. [^a-z]
means
“all character but lowercase letters”). Appending i
right after the right
bracket makes the match case-insensitive.
Match a parsing expression of a rule recursively and return its match result.
Match a subexpression and return its match result.
Match zero or more repetitions of the expression and return their match results in an array. The matching is greedy, i.e. the parser tries to match the expression as many times as possible. Unlike in regular expressions, there is no backtracking.
Match one or more repetitions of the expression and return their match results in an array. The matching is greedy, i.e. the parser tries to match the expression as many times as possible. Unlike in regular expressions, there is no backtracking.
Try to match the expression. If the match succeeds, return its match result,
otherwise return null
. Unlike in regular expressions, there is no
backtracking.
expression |count|
expression |min..max|
expression |count, delimiter-expression|
expression |min..max, delimiter-expression|
Match exact count
repetitions of expression
. If the match succeeds, return
their match results in an array.
-or-
Match expression at least min
but not more then max
times. If the match
succeeds, return their match results in an array. Both min
and max
may
be omitted. If min
is omitted, then it is assumed to be 0
. If max
is
omitted, then it is assumed to be infinity. Hence expression |..|
is an
equivalent of expression |0..|
and expression *
. expression |1..|
is
equivalent of expression +
.
If delimiter-expression
is specified, then it must present between each
expression
. Delimiters a not part of the result and just skipped.
count
, min
and max
must be positive integers or they can be names of
labels of prior visible expressions. In this case the labeled expressions
shall return non-negative integers which are considered as element counts.
If expressions to return something other, it is considered as 0
.
Try to match the expression. If the match succeeds, just return undefined
and
do not consume any input, otherwise consider the match failed.
Try to match the expression. If the match does not succeed, just return
undefined
and do not consume any input, otherwise consider the match failed.
This is a positive assertion. No input is consumed.
The predicate should be JavaScript code, and it's executed as a function. Curly braces in the predicate must be balanced.
The predicate should return
a boolean value. If the result is
truthy, the match result is undefined
, otherwise the match is
considered failed.
The predicate has access to all variables and functions in the Action Execution Environment.
This is a negative assertion. No input is consumed.
The predicate should be JavaScript code, and it's executed as a function. Curly braces in the predicate must be balanced.
The predicate should return
a boolean value. If the result is
falsy, the match result is undefined
, otherwise the match is
considered failed.
The predicate has access to all variables and functions in the Action Execution Environment.
Try to match the expression. If the match succeeds, return the matched text instead of the match result.
Match the expression and remember its match result under given label. The label must be a JavaScript identifier.
Labeled expressions are useful together with actions, where saved match results can be accessed by action's JavaScript code.
Match a sequence of expressions and return their match results in an array.
If the expression matches successfully, run the action, otherwise consider the match failed.
The action should be JavaScript code, and it's executed as a function. Curly braces in the action must be balanced.
The action should return
some value, which will be used as the
match result of the expression.
The action has access to all variables and functions in the Action Execution Environment.
Try to match the first expression, if it does not succeed, try the second one, etc. Return the match result of the first successfully matched expression. If no expression matches, consider the match failed.
Actions and predicates have these variables and functions available to them.
-
All variables and functions defined in the initializer at the beginning of the grammar are available.
-
Labels from preceding expressions are available as local variables, which will have the match result of the labelled expressions.
A label is only available after its labelled expression is matched:
rule = A:('a' B:'b' { /* B is available, A is not */ } )
A label in a sub-expression is only valid within the sub-expression:
rule = A:'a' (B: 'b') (C: 'b' { /* A and C are available, B is not */ })
-
options
is a variable that contains the parser options. -
error(message, where)
will report an error and throw an exception.where
is optional; the default is the value oflocation()
. -
expected(message, where)
is similar toerror
, but reportsExpected message but "other" found.
-
location()
returns an object like this:{ source: options.source, start: { offset: 23, line: 5, column: 6 }, end: { offset: 25, line: 5, column: 8 } }
For actions,
start
refers to the position at the beginning of the preceding expression, andend
refers to the position after the end of the preceding expression.For predicates,
start
andend
are the same, the location where the predicate is evaluated.offset
is a 0-based character index within the source text.line
andcolumn
are 1-based indices.Note that
line
andcolumn
are somewhat expensive to compute, so if you need location frequently, you might want to useoffset()
orrange()
instead. -
offset()
returns the start offset. -
range()
returns an array containing the start and end offsets, such as[23, 25]
. -
text()
returns the source text betweenstart
andend
(which will be "" for predicates).
Code fragments such as actions and predicates must have balanced curly braces, because pegjs doesn't parse the contents. It only looks at balanced braces to find the end of the code fragment.
If your code fragment needs an unbalanced brace in a string literal, you can balance it in a comment. For example:
brace = [{}] {
return text() === "{" ? 1 : -1; // } for balance
}
Syntax for defining attributes is similar to syntax of the Rust:
#[cached]
#[export]
rule = n:$[0-9]+ #[type(int)] { return parseInt(n); }
Attributes can have any name. Their sense completely is defined by a code of plug-ins which will process them. In an example above the attributes is used to mark the rule exported and caching parse result. Attribute for code block used to specify it return type, that useful for generating code for static typing languages. There are no built-in attributes.
Each attribute has name and value. In example above there three attributes with
names cached
, export
and type
. Attribute type
also has value int
. Attribute
value is written in the parenthesis at once after his name. There are no restrictions
for value of attribute except that the parentheses in it shall be balanced. In AST
all attributes live in the array property attributes
of nodes of type initializer
,
semantic_and
, semantic_or
or rule
. If you need the attributes attached to grammar,
attach them to the initialiser.
As described above, you can annotate your grammar rules with human-readable names that will be used in error messages. For example, this production:
integer "integer"
= digits:[0-9]+
will produce an error message like:
Expected integer but "a" found.
when parsing a non-number, referencing the human-readable name "integer." Without the human-readable name, PEG.js instead uses a description of the character class that failed to match:
Expected [0-9] but "a" found.
Aside from the text content of messages, human-readable names also have a subtler effect on where errors are reported. PEG.js prefers to match named rules completely or not at all, but not partially. Unnamed rules, on the other hand, can produce an error in the middle of their subexpressions.
For example, for this rule matching a comma-separated list of integers:
seq
= integer ("," integer)*
an input like 1,2,a
produces this error message:
Expected integer but "a" found.
But if we add a human-readable name to the seq
production:
seq "list of numbers"
= integer ("," integer)*
then PEG.js prefers an error message that implies a smaller attempted parse tree:
Expected end of input but "," found.
All comments which are present at grammar during parse get to a map under
a comments
key in a node with type grammar
. As a key to the comment serves
offset with which it begins. Each comment has following structure:
{
text: 'text in the comment, just after // or /* and before */',
multiline: true|false,// true for /**/ comments, false for // comments
location: location()
}
Both the parser generator and generated parsers should run well in the following environments:
- Node.js 4+
- Internet Explorer 9+
- Edge
- Firefox
- Chrome
- Safari
- Opera
PEG.js 2 is currently maintained by Mingun.
Original PEG.js is currently maintained by Futago-za Ryuu. Since it's inception in 2010, PEG.js was maintained by David Majda (@dmajda), until May 2017.
The Bower package is maintained by Michel Krämer (@michelkraemer).
- Project website (original repo)
- Wiki
- Source code
- Issue tracker
- Google Group (original repo)
- Stack Overflow (original repo)
- Twitter (original repo)
You are welcome to contribute code using GitHub pull requests. Unless your contribution is really trivial you should get in touch with me first (preferably by creating a new issue on the issue tracker) - this can prevent wasted effort on both sides.
Before submitting a pull request, please make sure you've checked out the Contribution Guidelines.
- Create a fork of https://github.com/Mingun/pegjs
- Clone your fork, and optionally create a new branch
- Run the command
npm install
from the root of your clone - Add and commit your changes
- Validate your changes:
- Lint the JavaScript changes (command line only, run
gulp lint
ornpm run lint
) - Run tests to ensure nothing's broken: see separate documentation
- Optionally, check benchmark results: see separate documentation
- Optionally, check commit impact (this is a js script, run
test/impact
)
- Lint the JavaScript changes (command line only, run
- If validation fails: reverse your commit, fix the problem and then add/commit again
- Push the commits from your clone to the fork
- From your fork, start a new pull request
It's also a good idea to check out the gulpfile.js that defines various tasks that are commented with a description of each task.
To see the list of contributors check out the repository's contributors page.