Skip to content
/ jpt Public

JSON Power Tool: Retrieve and manipulate JSON data using JSONPath, JSON Pointer, JSON Patch, and JSON Merge Patch. Written in Javascript, wrapped in shell, requiring jsc the JavaScriptCore binary.

License

Notifications You must be signed in to change notification settings

brunerd/jpt

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 

Repository files navigation

jpt

JSON Power Tool: Query and manipulate JSON and JSON Text Sequences using JSONPath, JSON Pointer, JSON Patch, and JSON Merge Patch.

Written in Javascript (ES5) and wrapped in a bit of shell, it can be used standalone or embedded in your bash/zsh scripts, requiring only jsc which has been standard on Macs since 10.4 and is widely available for Linux and even Windows with Linux subsystem installed.

It is a non-compiled script, so it can be easily studied, maintained, or modified. It avoids the need for code-signing and notarization on the Mac platform.

See my blog for articles, examples, and musing on the jpt: https://www.brunerd.com/blog/category/projects/jpt/

Examples

Please see the brunerd JSONPath GitHub page for an extensive list of example JSONPath queries.

JSONPath has a richly expressive syntax that can query the contents of JSON in ways JSON Pointer can't. For example filter expressions can find objects within an array:

% jpt '$.object_array[?(@.color == "blue")]' <<< '{"object_array":[{"id":2,"color":"red"},{"id":1,"color":"blue"}]}'
{
  "id": 1,
  "color": "blue"
}

The values within the found objects can be specifically reported:

% jpt '$.object_array[?(@.color == "blue")].id' <<< '{"object_array":[{"id":2,"color":"red"},{"id":1,"color":"blue"}]}'
1

JSON Pointer is extremely simple and can query a single property only, this uses the -T option to print a JSON string as text and not encoded within double quotes.

% jpt -T /object/array/1/mood <<< $'{"object":{"array":[{"mood":"excited"},{"mood":"intrigued"}]}}'
intrigued

Help File (-h)

jpt (v1.0.9) - JSON Power Tool (https://github.com/brunerd/jpt)

Usage:
jpt [options] [<query>] [<file>]

Arguments:
	[<query>] - JSONPath, JSON Pointer, or plutil-style keypath expression (optional), returns entire document otherwise.
	[<file>] - path to JSON file

Notes:
	Default output mode is JSON (RFC8259/STD90)
	Multiple JSON texts will be output as a JSON Text Sequence (RFC 7464) (use -M to change behavior and processing)
	jpt accepts input by file redirection, Unix pipe, here string, heredoc and via /dev/stdin (end with Control-D)
	jpt accepts JSON, JSON text Sequences, and non-JSON mutations such as JSON5, NDJSON and jpt's own JSONPath Object Literals
	JSON Text Sequences will be strictly parsed to RFC 7464 specs, non-JSON elements are only tolerated in single and concatenated JSON texts
	Non-JSON elements are corrected automatically, if possible, before parsing (use -c for comments via stderr with an exit status of 1)
	  JSON5 Rehabilitations (mostly):
	    Unquoted object keys and single quoted strings are converted to JSON strings
	    Additional Unicode whitespaces and paragraph separators are converted to normal spaces (0x20) and newlines (0x0A)
	    Trailing commas are removed from object and array elements
	    Explicit plus signs on numbers are removed, decimals without leading zeroes are restored
	    Multi-line strings with escaped line breaks are reverted to standard JSON strings
	      Literal tabs in strings are removed also
	    Escaped character conversions: 
		  \0 to null
		  \v to '\u000c'
	    JSON5 number type conversions: 
		  NaN to null (use -8 to convert to "NaN" string)
		  Inifinity to null (use -I to convert to "Infinity" string, with sign if specified)
	    Single-line (//) and multi-line (/* */) style comments are removed
		  Additionally multi-line XML (<!-- -->) and single line Shell (#), ASM (;), and Lua (--) comments are removed
	
General Options:
	-h this help screen

JSON Output Options:
	-i "<number/string>" indent number of spaces (0-10) or use a <string> for each level of indent
	-O Order property names in objects alphabetically
	-u Unicode escape (\u) all characters above 0x7E

Advanced JSON Output Options:
	-a always output result in an array
	-c comment on corrections to non-standard JSON via stderr output (with exit status of 1)
	-F Flatten array output
	-I Convert JSON5 +/-Infinity value to a string with signedness (otherwise converts to null)
	-N Nested single element arrays are reduced
	-8 Convert JSON5 NaN to a string (otherwise converts to null)
	-/ Escape solidus / with a reverse solidus like \/, useful for HTML with embedded JSON encoded HTML

Input Options:
	-D Detect truncation within objects and arrays in concatenated JSON only
	-g path-only JSONPath Object Literals are assigned a value of null ("gratis nulls")
	-G Guard previously defined JSONPath Object Literals from being modified by new statements
	-H Help truncated JSON by closing up open strings, objects and arrays
	-S Treat input as a JSON string

Multiple JSON Text Options:
	-M "<value>" - options for working with multiple JSON texts
		S - Output JSON Text Sequences strictly conforming to RFC 7464 (default)
		N - Output newline delimited JSON texts
		C - Output concatenated JSON texts
		A - Gather JSON texts into an array, post-query and post-patching
		a - Gather JSON texts into an array, pre-query and pre-patching 

Alternate Output Modes (non-JSON):
	-l output the length of the resulting array, object property count, or length of string

	-J JSONPath path(s) of the object returned by the query
	-j JSONPath path(s) matched by the query results

	-R JSON Pointer path(s) of the object returned by the query
	-r JSON Pointer path(s) matched by the query results
		Specifiying -EW or -Ew encodes both -r and -R JSON Pointer paths in URI fragment style

	  -J and -R path output options:
		-C append the "constructor" type (Array, Object, String, Boolean, Number, or null)
		-K/-k print property (key) names and array indices only with no preceding path, default indent is 2
			-i "<value>" indent spaces (0-10) or character for each level of indent

	-L JSONPath Object Literal notation output of the resulting object
		JPOL Format: <JSONPath expression>=<JSON value>
			<JSONPath expression> is simply Javascript expression syntax with $ as the object name
				Example: $.key, $["key"], $['key'], or $[0]
			<JSON value> is any valid JSON value (plus single quoted strings)
				Example: 'string', "string", [], {}, 42, true, false, null

	  JSONPath output options for -L -J and -j:
		-d Use dot notation for object property names when possible, rather than bracket notation
		-q Use single quotes for bracketed property names, string values remain JSON double quoted
		-Q Use single quotes for BOTH bracketed property names AND string values (-L only)
		-u encode characters above 7E with \u escape

	  Output limiting options for -L, -J, and -R:
		-P Only print Primitive data types (String, Boolean, Number, null) omitting Arrays and Objects
		-Z "<int>" Depth
			Combined with -L it will coalesce lower depth levels into a compound JSON object/array statement
			Combined with -P both -J and -R will return only purely primitive nodes at or below the Z level

	-T textual output of all data (omits property names and indices)
	  Options:
		-e encode whitespace and escapes: \b \f \n \r \t \v and \\ (\ escaped formats only)
		-i "<value>" indent spaces (0-10) or character string for each level of indent
		-n convert null value to string 'null' (pre-encoding)

		-E "<value>" encoding options for -T output:

		  Encodes string characters below 0x20 and above 0x7E with pass-through for all else:
			x 	"\x" prefixed hexadecimal UTF-8 strings
			O 	"\nnn" style octal for UTF-8 strings
			0 	"\0nnn" style octal for UTF-8 strings
			u 	"\u" prefixed Unicode for UTF-16 strings
			U 	"\U "prefixed Unicode Code Point strings
			E 	"\u{...}" prefixed ES2016 Unicode Code Point strings
			W 	"%nn" Web encoded UTF-8 string using encodeURI (respects scheme and domain of URL)
			w 	"%nn" Web encoded UTF-8 string using encodeURIComponent (encodes all components URL)

			  -W encodes whitespace characters also
			  -A encodes ALL characters
		
		  Encodes both strings and numbers with pass-through for all else:
			h 	"0x" prefixed lowercase hexadecimal, UTF-8 strings
			H 	"0x" prefixed uppercase hexadecimal, UTF-8 strings
			o 	"0o" prefixed octal, UTF-8 strings
			6 	"0b" prefixed binary, 16 bit _ spaced numbers and UTF-16 strings
			B 	"0b" prefixed binary, 8 bit _ spaced numbers and UTF-16 strings
			b 	"0b" prefixed binary, 8 bit _ spaced numbers and UTF-8 strings

			  -U whitespace is left untouched (not encoded)

Data Processing:
	jpt can apply JSON Patch (RFC 6902), JSON Merge Patch (RFC 7396), and other operations to JSON text(s)

	JSON Patch processing:
		-X "<filepath to JSON Patch>" a file path to a JSON Patch array (RFC 6902)
		-x "<inline JSON Patch>" inline JSON Patch text (RFC 6902)
		
	JSON Patch Example (from RFC 6902):
		 [
			 { "op": "test", "path": "/a/b/c", "value": "foo" },
			 { "op": "remove", "path": "/a/b/c" },
			 { "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
			 { "op": "replace", "path": "/a/b/c", "value": 42 },
			 { "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
			 { "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
		]

	Single operations for JSON Patch, JSON Merge Patch, and others can be crafted using -o plus -p (path) and possibly -f (from) or -v/-V (value)

		-o "<operation>":
		 [JSON Patch Operations]
		  add
			-o add -p "<path>" -v "<value>"
				add/replace value in object or insert value into array index (RFC 6902)
		  replace
			-o replace -p "<path>" -v "<value>"
				replace value of existing object property (key) or array index (RFC 6902)
		  remove
			-o remove -p "<path>"
				remove specified path from the document (RFC 6902)
		  move
			-o move -f "<from>" -p "<path>"
				move a path to a new or existing path/node(RFC 6902)
		  copy
			-o copy -f "<from>" -p "<path>"
				copy a path to a new or existing path/node (RFC 6902)
		  test
			-o test -p "<path>" -v "<value>"
				test if a path matches a value exactly (RFC 6902), return 0 if true, 1 if false

		 [Other Operations]
		  diff
			-o diff -v "<value>"
				Given a value, compare with JSON text and produce a JSON Merge Patch (RFC 7396) document
		  mergepatch
			-o mergepatch (-v/-V "<value>" || -f "<path>") [-p <path>]
				JSON Merge Patch (RFC 7396) operation: null values delete target object property
				Can pull data from -v/-V "<value>" OR -f "<path>", can merge to specific -p "<path>" (JSONPaths allows multiple)
		  merge
			-o merge (-v/-V "<value>" || -f "<path>") [-p "<path>"]
				Non-RFC 7396 merging operation, object properties with null values are NOT removed
				Can pull data from -v/-V "<value>" or -f "<path>", can merge to specific -p "<path>" (JSONPaths allows multiple)
		  merge0
			-o merge0 (-v/-V "<value>" || -f "<path>") [-p "<path>"]
				Merge object properties that DO NOT intersect with source JSON
				Can pull data from -v/-V "<value>" or -f "<path>", can merge to specific -p "<path>" (JSONPaths allows multiple)
		  merge1
			-o merge1 (-v/-V "<value>" || -f "<path>") [-p "<path>"]
				Merge object properties that intersect with source JSON
				Can pull data from -v/-V "<value>" or -f "<path>", can merge to specific -p "<path>" (JSONPaths allows multiple)
				
		-p "<path>":
			-p "<path>" the target path expressed in JSON Pointer (RFC 6901) or JSONPath

		-f "<from path>":
			copy and move require a JSON Pointer (RFC 6901) or JSONPath expression

		-v/-V "<value>":
			-v "<JSON value>" is a single JSON text
			-V "<filespec to JSON value>" is a file path to a single JSON text
		  Options for -v/-V:
			-s treat input as a string
				
JSONPath Primer
	$
		the root of the JSON document, ALL JSONPath QUERIES MUST BEGIN WITH $
	.key or ['key']
		a "child" operator for an object property named 'key' in a JSON object
	..key or ..['key']
		is a recursive operator that will find all properties named 'key' within the object
	['key name'] or ["key \ud83d\udd11"]
		property names with spaces or escaped values MUST use bracket notation
	.* or [*]
		* returns the values of all the keys within an object or all indices in an array
	[start:stop:step]
		slice operation for arrays, accepts pos/neg integers, script and filter expressions, or leave empty
	[?(@.id >= 1 && @.id <= 10 )] or [?(@.key == "string" && !@['key2'])] or the off-spec [?(@name =~ /key.*/)], etc...
		filter expressions can return one or more matching objects within an array
		@ is the current object, dot and bracket notation can be used to query child nodes
		Use logical operators like: == (equal), != (not equal), > (greater than), < (less than), >= (greater or equal), <= (less or equal)
		Multiple criteria can be evaluated with && (AND) and || (OR). Regular expressions can also be used: '=~ /regexp/'
		You can negate a value using !
		@.length is the length of an array or string
		@name is the current property name (an off-spec but useful quirk of the original Goessner code)
	[(@.length/2)]
		"script expression", returns a single value, use as an array index selector that allow division
		Note: These will NOT make it into the IETF JSONPath spec
	[1] or [-1]
		array index, integers only, positive starts at the beginning, negative references from the end 
	["-"]
		The nonexistent array member after the last array element
		Use with data alteration operations, will always be empty for queries of arrays
		Akin to /- in JSON Pointer, if used with an object simply looks for a key named "-"
	["a","b"]
		union, a comma separated list of expressions values of multiple properties at the same level in an object
		Unions allow for: quoted property names (single or double), numbers, *, filter and script expressions, and slices

	For more examples: https://github.com/brunerd/jsonpath
	
JSON Pointer (RFC 6901) Primer
	""		an empty string represents the JSON document
	/		is the root or next child node with a property named "" an empty string (JSONPath $[""])
	/key		property named key in an object
	/1		property named 1 in an object or a numeric index in an array
	/-		the nonexistent array member after the last array element, use with data alteration operations
	~		in a property name must be escaped as ~0
	/		in a property name must be escaped as ~1

	Example: /JSON pointer/does/this/1/thing/well

keypath primer:
	This arcane summoning hails from ye olde NextStep and still used by plutil
	
	.		a period is used to delimit key names, 
				literal periods can be backslash (\) escaped
	key		an object name or array element, there is no root character
								
	Keypath Example: this.is.1.ugly.key\.path

	In JSON Pointer: /this/is/1/ugly/key.path
	And in JSONPath: $.this.is[1].ugly.["key.path"]

	Note: If a keypath query begins with characters that collide with JSONPath ($), jq-style (. or [) or JSON Pointer (/) it will be evaluated as one of those.

Requirements:

bash or zsh on a supported OS:

  • macOS 10.4+
  • Linux with jsc
  • Windows with Linux subsystem and jsc

About

JSON Power Tool: Retrieve and manipulate JSON data using JSONPath, JSON Pointer, JSON Patch, and JSON Merge Patch. Written in Javascript, wrapped in shell, requiring jsc the JavaScriptCore binary.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages