This package adds a command [json] to the interpreter, and defines a new Tcl_Obj type to store the parsed JSON document. The [json] command directly manipulates values whose string representation is valid JSON, in a similar way to how the [dict] command directly manipulates values whose string representation is a valid dictionary. It is similar to [dict] in performance.
Also provided is a command [json template] which generates JSON documents by interpolating values into a template from a supplied dictionary or variables in the current call frame. The templates are valid JSON documents containing string values which match the regex "^~[SNBJTL]:.+$". The second character determines what the resulting type of the substituted value will be:
- S: A string.
- N: A number.
- B: A boolean.
- J: A JSON fragment.
- T: A JSON template (substitutions are performed on the inserted fragment).
- L: A literal - the resulting string is simply everything from the forth character onwards (this allows literal strings to be included in the template that would otherwise be interpreted as the substitutions above).
None of the first three characters for a template may be escaped.
The value inserted is determined by the characters following the substitution type prefix. When interpolating values from a dictionary they name keys in the dictionary which hold the values to interpolate. When interpolating from variables in the current scope, they name scalar or array variables which hold the values to interpolate. In either case if the named key or variable doesn't exist, a JSON null is interpolated in its place.
- [json get json_val ?key ... ?modifier??] - Extract the value of a portion of the json_val, returns the closest native Tcl type (other than JSON) for the extracted portion.
- [json parse json_val] - A deprecated synonym for [json get json_val].
- [json get_typed json_val ?key ... ?modifier??] - Extract the value of a portion of the json_val, returns a two element list: the first being the value that would be returned by [json get] and the second being the JSON type of the extracted portion.
- [json extract json_val ?key ... ?modifier??] - Extract the value of a portion of the json_val, returns the JSON fragment.
- [json exists json_val ?key ... ?modifier??] - Tests whether the supplied key path and modifier resolve to something that exists in json_val
- [json set json_variable_name ?key ...? value] - Updates the JSON value stored in the variable json_variable_name, replacing the value referenced by key ... with the JSON value value.
- [json unset json_variable_name ?key ...?] - Updates the JSON value stored in the variable json_variable_name, removing the value referenced by key ...
- [json normalize json_val] - Return a "normalized" version of the input json_val - all optional whitespace trimmed.
- [json template json_val ?dictionary?] - Return a JSON value by interpolating the values from dictionary into the template, or from variables in the current scope if dictionary is not supplied, in the manner described above.
- [json new type value] - Return a JSON fragment of type type and value value.
- [json fmt type value] - A deprecated synonym for [json new type value].
- [json foreach varlist1 json_val1 ?varlist2 json_val2 ...? script] - Evaluate script in a loop in a similar way to the [foreach] command. In each iteration, the values stored in the iterator variables in varlist are the JSON fragments from json_val. Supports iterating over JSON arrays and JSON objects. In the JSON object case, varlist must be a two element list, with the first specifiying the variable to hold the key and the second the value. In the JSON array case, the rules are the same as the [foreach] command.
- [json lmap varlist1 json_val1 ?varlist2 json_val2 ...? script] - As for [json foreach], except that it is collecting - the result from each evaluation of script is added to a list and returned as the result of the [json lmap] command. If the script results in a TCL_CONTINUE code, that iteration is skipped and no element is added to the result list. If it results in TCL_BREAK the iterations are stopped and the results accumulated so far are returned.
- [json pretty json_val] - Returns a pretty-printed string representation of json_val. Useful for debugging or inspecting the structure of JSON data.
The commands [json get], [json get_typed], [json extract] and [json exists] accept a path specification that names some subset of the supplied json_val. The rules are similar to the equivalent concept in the [dict] command, except that the paths used by [json] allow indexing into JSON arrays by the integer key (or a string matching the regex "^end(-[0-9]+)?$"), and that the last element can be a modifier:
- ?type - Returns the type of the named fragment.
- ?length - When the path refers to an array, this returns the length of the array. When the path refers to a string, this returns the number of characters in the string. All other types throw an error.
- ?size - Valid only for objects, returns the number of keys defined in the object.
- ?keys - Valid only for objects, returns a list of the keys in the object.
A literal value that would match one of the above modifiers can be used as the last element in the path by doubling the ?:
json get {
{
"foo": {
"?size": "quite big"
}
}
} foo ??size
Returns "quite big"
Produce a JSON value from a template:
json template {
{
"thing1": "~S:val1",
"thing2": ["a", "~N:val2", "~S:val2", "~B:val2", "~S:val3", "~L:~S:val1"],
"subdoc1": "~J:subdoc",
"subdoc2": "~T:subdoc"
}
} {
val1 hello
val2 1e6
subdoc {
{ "thing3": "~S:val1" }
}
}
Result:
{"thing1":"hello","thing2":["a",1000000.0,"1e6",true,null,"~S:val1"],"subdoc1":{"thing3":"~S:val1"},"subdoc2":{"thing3":"hello"}}
Good performance was a requirement for rl_json, because it is used to handle large volumes of data flowing to and from various JSON based REST apis. It's generally the fastest option for working with JSON values in Tcl from the options I've tried, with the next closest being yajltcl. These benchmarks report the median times in microseconds, and produce quite stable results between runs. Benchmarking was done on a MacBook Air running Ubuntu 14.04 64bit, Tcl 8.6.3 built with -O3 optimization turned on, and using an Intel i5 3427U CPU.
This benchmark compares the relative performance of extracting the field containing the string "obj" from the JSON doc:
{
"foo": "bar",
"baz": ["str", 123, 123.4, true, false, null, {"inner": "obj"}]
}
The compared methods are:
Name | Notes | Code |
---|---|---|
old_json_parse | Pure Tcl parser | dict get [lindex [dict get [json_old parse [string trim $json]] baz] end] inner |
rl_json_parse | dict get [lindex [dict get [json parse [string trim $json]] baz] end] inner |
|
rl_json_get | Using the built-in accessor method | json get [string trim $json] baz end inner |
yajltcl | dict get [lindex [dict get [yajl::json2dict [string trim $json]] baz] end] inner |
|
rl_json_get_native | json get $json baz end inner |
The use of [string trim $json] is to defeat the caching of the parsed representation, forcing it to reparse the string each time since we're measuring the parse performance here. The exception is the rl_json_get_native test which demonstrates the performance of the cached case.
-- parse-1.1: "Parse a small JSON doc and extract a field" --------------------
| This run
old_json_parse | 241.595
rl_json_parse | 5.540
rl_json_get | 4.950
yajltcl | 8.800
rl_json_get_native | 0.800
This benchmark compares the relative performance of various ways of dynamically generating a JSON document. Although all the methods produce the same string, only the "template" and "template_dict" variants handle nulls in the general case - the others manually test for null only for the one field that is known to be null, so the performance of these variants would be worse in a real-world scenario where all fields would need to be tested for null.
The JSON doc generated in each case is the one produced by the following JSON template (where a(not_defined) does not exist and results in a null value in the produced document):
{
"foo": "~S:bar",
"baz": [
"~S:a(x)",
"~N:a(y)",
123.4,
"~B:a(on)",
"~B:a(off)",
"~S:a(not_defined)",
"~L:~S:not a subst",
"~T:a(subdoc)",
"~T:a(subdoc2)"
]
}
The produced JSON doc is:
{"foo":"Bar","baz":["str\"foo\nbar",123,123.4,true,false,null,"~S:not a subst",{"inner":"Bar"},{"inner2":"Bar"}]}
The code for these variants are too long to include in this table, refer to bench/new.bench for the details.
Name | Notes |
---|---|
old_json_fmt | Pure Tcl implementation, builds JSON from type-annotated Tcl values |
rl_json_new | rl_json's [json new], API compatible with the pure Tcl version used in old_json_fmt |
template | rl_json's [json template] |
yajltcl | yajltcl's type-annotated Tcl value approach |
template_dict | As for template, but using a dict containing the values to substitute |
yajltcl_dict | As for yajltcl, but extracting the values from the same dict used by template_dict |
-- new-1.1: "Various ways of dynamically assembling a JSON doc" ---------------
| This run
old_json_fmt | 49.450
rl_json_new | 10.240
template | 4.520
yajltcl | 7.700
template_dict | 2.500
yajltcl_dict | 7.530
Older versions used the yajl c library to parse the JSON string and properly quote generated strings when serializing JSON values, but currently a custom built parser and string quoter is used, removing the libyajl dependency. JSON values are parsed to an internal format using Tcl_Objs and stored as the internal representation for a new type of Tcl_Obj. Subsequent manipulation of that value use the internal representation directly.
Copyright 2015 Ruby Lane. Licensed under the same terms as the Tcl core.