SqlJsonSurfer

This work is based on https://github.com/jsurfer/JsonSurfer and modified to support the JSONPath/SQL standard. The current implementation is a work in progress, it doesn't cover all JSONPath/SQL features.

Most notable changes:

support for strict/lax modes in the syntax, but only lax mode is currently supported (one can specify lax in the JSONPath expression).
array range syntax is [a to b], where b is inclusive.
properties are quoted as $."my property", instead of $['my property'].
strings can be only double-quoted, single-quoted strings are not supported.
for regex filter, SQL uses SQL's like_regex operator

JsonSurfer - Let's surf on Json!

This repo was copied from https://github.com/jsurfer/JsonSurfer

Why JsonSurfer

Streaming

No need to deserialize entire json into memory.
JsonPath

Selectively extract json data by the power of JsonPath.
Stoppable

JsonSurfer is built on stoppable SAX-like interface that allows the processor to stop itself if necessary.
Non-Blocking

JsonSurfer is event-driven and offers non-blocking parser interface.
Binary format

Support multiple binary data formats including Avro, CBOR, Protobuf, Smile and Ion.

Getting started

What is JsonPath?

Supported JsonPath operator in JsonSurfer:

Operator	Description
`$`	root
`@`	current node
`*`	wildcard
`..`	recursive descent
`.<name>`	child
`['<name>' (, '<name>')]`	child/children
`[<number> (, <number>)]`	index/indices
`?(<expression>)`	filter expression

JsonSurfer is available in central maven repository.

JsonSurfer has drivers for most popular json libraries including: Gson, Jackson, FastJson and JsonSimple. Choose one and add to your POM.

<dependency>
    <groupId>com.hazelcast.jsurfer</groupId>
    <artifactId>jsurfer-gson</artifactId>
    <version>0.10</version>
</dependency>

<dependency>
    <groupId>com.hazelcast.jsurfer</groupId>
    <artifactId>jsurfer-jackson</artifactId>
    <version>0.10</version>
</dependency>

<dependency>
    <groupId>com.hazelcast.jsurfer</groupId>
    <artifactId>jsurfer-fastjson</artifactId>
    <version>0.10</version>
</dependency>

<dependency>
    <groupId>com.hazelcast.jsurfer</groupId>
    <artifactId>jsurfer-jsonsimple</artifactId>
    <version>0.10</version>
</dependency>

Usage:

Create your JsonSurfer:

JsonSurfer has flexible constructor. You can create yourself or pick a prebuilt one according the json library you used:

Gson

        // use gson parser and use gson provider use to deserialize json into gson model i.e.com.google.gson.JsonElement
        JsonSurfer surfer = new JsonSurfer(GsonParser.INSTANCE, GsonProvider.INSTANCE);

or

        JsonSurfer surfer = JsonSurferGson.INSTANCE;

Jackson

        JsonSurfer surfer = new JsonSurfer(JacksonParser.INSTANCE, JacksonProvider.INSTANCE);

or

        JsonSurfer surfer = JsonSurferJackson.INSTANCE;

JsonSimple

        // use json-simple parser and json-simple provider to deserialize json into json-simple model i.e.org.json.simple.JSONObject or org.json.simple.JSONArray
        JsonSurfer surfer = new JsonSurfer(JsonSimpleParser.INSTANCE, JsonSimpleProvider.INSTANCE);

or

        JsonSurfer surfer = JsonSurferJsonSimple.INSTANCE;

Fastjson

        JsonSurfer surfer = new JsonSurfer(FastJsonParser.INSTANCE, FastJsonProvider.INSTANCE);

or

        JsonSurfer surfer = JsonSurferFastJson.INSTANCE;

Collect value by JsonPath

        Collector collector = surfer.collector(sample);
        ValueBox<String> box1 = collector.collectOne("$.store.book[1].category", String.class);
        ValueBox<Object> box2 = collector.collectOne("$.store.book[2].isbn");
        ValueBox<Collection<Object>> box3 = collector.collectAll("$.store.book[*]");
        collector.exec(); // make sure exec() invoked before getting value from boxes
        box1.get();
        box2.get();
        box3.get();

"Surfing" in Json and collecting matched value in the listeners

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$.store.book[*]", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Reuse listener binding.

SurfingConfiguration is thread-safe as long as your listeners are stateless.

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        SurfingConfiguration config = surfer.configBuilder()
                .bind("$.store.book[*]", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .build();        
        surfer.surf(sample1, config);
        surfer.surf(sample2, config);

Compiled JsonPath

JsonPath object is immutable and can be reused safely.

Tips: Most of JsonSurfer API have two version: accepting raw JsonPath string or JsonPath object. The latter is always faster than the former without compiling JsonPath.

        JsonPath compiledPath = JsonPathCompiler.compile("$..book[1,3]['author','title']");
        String value = surfer.collectOne(read("sample.json"), String.class, compiledPath);

JsonPath Filters

Filter operators

Operator	Description
==	equal
<	less than
>	greater than

You can use logical operators '&&' and '||' to create more complex filter expression. For example:

$.store.book[*]?(@.price < 10 || @.category && @.isbn && @.price>10).volumes[*]?(@.chapter == 1)

Resolver API:

Limitation: Wildcard and Recursive Descent are NOT supported.
As of 1.2.6, JsonSurfer provides another way of processing json. You can directly resolve value with JsonPath from a well-built DOM like HashMap or even POJO:

        Book book = new Book();
        book.setAuthor("Leo");
        book.setCategory("Fiction");
        book.setPrice(100.0d);
        book.setTitle("JsonSurfer is great!");
        System.out.print(compile("$.author").resolve(book, new PoJoResolver()));

which prints "Leo".

        List<String> list = Arrays.asList("foo", "bar");
        HashMap<String, Object> map = new HashMap<String, Object>();
        map.put("list", list);
        System.out.println(compile("$.list[1]").resolve(map, JavaCollectionProvider.INSTANCE));

which prints "bar".

If you want to process POJO with full JsonPath feature, you can convert the POJO into binary format and then surfer on it.

Binaray format (Jackson only)

By importing Jackson binary format backend, JsonSurfer is capable to surfer with multiple binary object representation formats such as Avro, CBOR, Protobuf(A known bug to be fixed in Jackson 2.9.6), Smile and Ion.

For example, if you want to surfer with CBOR data, firstly, CBOR format backend need to be imported as dependency.

    <dependency>
        <groupId>com.fasterxml.jackson.dataformat</groupId>
        <artifactId>jackson-dataformat-cbor</artifactId>
        <version>${jackson.version}</version>
    </dependency>

Then create a JsonSurfer with CBOR-backed JacksonParser and surfer as usual

    surfer = new JsonSurfer(new JacksonParser(new CBORFactory()), provider);

Find more examples here: https://github.com/hazelcast/JsonSurfer/blob/master/jsurfer-all/src/test/java/org/jsfr/json/JacksonParserTest.java

Share data among processors

Since JsonSurfer emit data in the way of callback, it would be difficult if one of your processing depends on another. Therefore, a simple transient map is added for sharing data among your processors. Following unit test shows how to use it:

        surfer.configBuilder().bind("$.store.book[1]", new JsonPathListener() {
            @Override
            public void onValue(Object value, ParsingContext context) {
                context.save("foo", "bar");
            }
        }).bind("$.store.book[2]", new JsonPathListener() {
            @Override
            public void onValue(Object value, ParsingContext context) {
                assertEquals("bar", context.load("foo", String.class));
            }
        }).bind("$.store.book[0]", new JsonPathListener() {
            @Override
            public void onValue(Object value, ParsingContext context) {
                assertNull(context.load("foo", String.class));
            }
        }).buildAndSurf(read("sample.json"));

Control parsing

How to pause and resume parsing.

    SurfingConfiguration config = surfer.configBuilder()
            .bind("$.store.book[0]", new JsonPathListener() {
                @Override
                public void onValue(Object value, ParsingContext context) {
                    LOGGER.info("The first pause");
                    context.pause();
                }
            })
            .bind("$.store.book[1]", new JsonPathListener() {
                @Override
                public void onValue(Object value, ParsingContext context) {
                    LOGGER.info("The second pause");
                    context.pause();
                }
            }).build();
    ResumableParser parser = surfer.getResumableParser(read("sample.json"), config);
    assertFalse(parser.resume());
    LOGGER.info("Start parsing");
    parser.parse();
    LOGGER.info("Resume from the first pause");
    assertTrue(parser.resume());
    LOGGER.info("Resume from the second pause");
    assertTrue(parser.resume());
    LOGGER.info("Parsing stopped");
    assertFalse(parser.resume());

Completely stop parsing. Refer to Stoppable parsing

Java 8 Streams API support

As of 1.4, JsonSurfer can create an iterator from Json and JsonPath. Matched value can be pulled from the iterator one by one without loading entire json into memory.

    Iterator iterator = surfer.iterator(read("sample.json"), JsonPathCompiler.compile("$.store.book[*]"));

Java8 user can also convert the iterator into a Stream

    Stream<Object> targetStream = StreamSupport.stream(
          Spliterators.spliteratorUnknownSize(iterator, Spliterator.ORDERED),
          false);

Functions implemented by Streams API

    public static Stream<Object> toStream(String json, String path) {
        Iterator<Object> iterator = JsonSurferJackson.INSTANCE.iterator(json, JsonPathCompiler.compile(path));
        return StreamSupport.stream(Spliterators.spliteratorUnknownSize(iterator, Spliterator.ORDERED), false);
    }

    public static void main(String[] s) throws Exception {
        String json = "[1,3,5,7,11]";
        // Count
        System.out.println(toStream(json, "$[*]").count());
        // Max
        toStream(json, "$[*]").mapToInt(o -> ((LongNode) o).asInt()).max().ifPresent(System.out::println);
        // Min
        toStream(json, "$[*]").mapToInt(o -> ((LongNode) o).asInt()).min().ifPresent(System.out::println);
        // Average
        toStream(json, "$[*]").mapToDouble(o -> ((LongNode) o).asDouble()).average().ifPresent(System.out::println);
    }

Non-Blocking parsing

As of 1.4, JsonSurfer support non-blocking parsing for JacksonParser. You can achieve 100% non-blocking JSON processing with JsonSurfer in a NIO application. Let's take a Vertx request handler as an example:

    Vertx vertx = Vertx.vertx();
    HttpServer server = vertx.createHttpServer(new HttpServerOptions());
    JsonSurfer surfer = JsonSurferJackson.INSTANCE;
    SurfingConfiguration config = surfer.configBuilder()
            .bind("$[*]", (JsonPathListener) (value, context) -> {
                // Handle json
                System.out.println(value);
            }).build();
    server.requestHandler(request -> {
        NonBlockingParser parser = surfer.createNonBlockingParser(config);
        request.handler(buffer -> {
            byte[] bytes = buffer.getBytes();
            System.out.println("Received " + bytes.length + " bytes");
            parser.feed(bytes, 0, bytes.length);
        });
        request.endHandler(aVoid -> {
            parser.endOfInput();
            System.out.println("End of request");
            request.response().end();
        });
    }).listen(8080);

Examples

Sample Json:

{
    "store": {
        "book": [
            {
                "category": "reference",
                "author": "Nigel Rees",
                "title": "Sayings of the Century",
                "price": 8.95
            },
            {
                "category": "fiction",
                "author": "Evelyn Waugh",
                "title": "Sword of Honour",
                "price": 12.99
            },
            {
                "category": "fiction",
                "author": "Herman Melville",
                "title": "Moby Dick",
                "isbn": "0-553-21311-3",
                "price": 8.99
            },
            {
                "category": "fiction",
                "author": "J. R. R. Tolkien",
                "title": "The Lord of the Rings",
                "isbn": "0-395-19395-8",
                "price": 22.99
            }
        ],
        "bicycle": {
            "color": "red",
            "price": 19.95
        }
    },
    "expensive": 10
}

JsonPath	Result
`$.store.book[*].author`	Find the authors of all books
`$..author`	All authors
`$.store.*`	All things in store
`$.store..price`	The price of everything in the store
`$..book[2]`	The third book
`$..book[0,1]`	The first two books
`$.store.book[*]?(@.price==8.95)`	Filter all books whose price equals to 8.95
`$.store.book[*]?(@.category=='fiction')`	Filter all books which belong to fiction category
`$.store.book[*]?(@.author like_regex /tolkien/i)`	All books matching regex

Find the authors of all books:

$.store.book[*].author

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$.store.book[*].author", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

"Nigel Rees"
"Evelyn Waugh"
"Herman Melville"
"J. R. R. Tolkien"

All authors

$..author

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$..author", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

"Nigel Rees"
"Evelyn Waugh"
"Herman Melville"
"J. R. R. Tolkien"

All things in store

$.store.*

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$.store.*", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

[{"category":"reference","author":"Nigel Rees","title":"Sayings of the Century","price":8.95},{"category":"fiction","author":"Evelyn Waugh","title":"Sword of Honour","price":12.99},{"category":"fiction","author":"Herman Melville","title":"Moby Dick","isbn":"0-553-21311-3","price":8.99},{"category":"fiction","author":"J. R. R. Tolkien","title":"The Lord of the Rings","isbn":"0-395-19395-8","price":22.99}]
{"color":"red","price":19.95}

The price of everything in the store

$.store..price

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$.store..price", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

The third book

$..book[2]

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$..book[2]", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

{"category":"fiction","author":"Herman Melville","title":"Moby Dick","isbn":"0-553-21311-3","price":8.99}

The first two books

$..book[0,1]

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$..book[0,1]", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

{"category":"reference","author":"Nigel Rees","title":"Sayings of the Century","price":8.95}
{"category":"fiction","author":"Evelyn Waugh","title":"Sword of Honour","price":12.99}

Filter all books whose price equals to 8.95

$.store.book[*]?(@.price==8.95)

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$.store.book[*]?(@.price==8.95)", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

{"category":"reference","author":"Nigel Rees","title":"Sayings of the Century","price":8.95}

Filter all books which belong to fiction category

$.store.book[*]?(@.category=='fiction')

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$.store.book[*]?(@.category=='fiction')", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

{"category":"fiction","author":"Evelyn Waugh","title":"Sword of Honour","price":12.99}
{"category":"fiction","author":"Herman Melville","title":"Moby Dick","isbn":"0-553-21311-3","price":8.99}
{"category":"fiction","author":"J. R. R. Tolkien","title":"The Lord of the Rings","isbn":"0-395-19395-8","price":22.99}

All books matching regex

$.store.book[*]?(@.author like_regex "tolkien")

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$.store.book[*]?(@.author like_regex \"tolkien\")')", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .buildAndSurf(sample);

Output

{"author":"J. R. R. Tolkien","price":22.99,"isbn":"0-395-19395-8","category":"fiction","title":"The Lord of the Rings"}

Stoppable parsing

The parsing is stopped when the first book found and printed.

$..book[0,1]

        JsonSurfer surfer = JsonSurferGson.INSTANCE;
        surfer.configBuilder()
                .bind("$..book[0,1]", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {                        
                        System.out.println(value);
                        context.stop();
                    }
                })
                .buildAndSurf(sample);

Output

{"category":"reference","author":"Nigel Rees","title":"Sayings of the Century","price":8.95}

Benchmark

JsonSurfer is fast !!! The benchmark is powered by JMH

Benchmark                                                       Mode  Cnt       Score       Error  Units
BenchmarkCollectSingleValue.benchmarkFastjson                  thrpt   10  139772.275      8854.369  ops/s
BenchmarkCollectSingleValue.benchmarkFastjsonWithJsonSurfer    thrpt   10  699176.961      23396.619  ops/s
BenchmarkCollectSingleValue.benchmarkGson                      thrpt   10  139394.358      6019.764  ops/s
BenchmarkCollectSingleValue.benchmarkGsonWithJsonSurfer        thrpt   10  632155.657      15484.499  ops/s
BenchmarkCollectSingleValue.benchmarkJackson                   thrpt   10  160545.079      7006.525  ops/s
BenchmarkCollectSingleValue.benchmarkJacksonWithJsonSurfer     thrpt   10  451870.586      13132.576  ops/s
BenchmarkCollectSingleValue.benchmarkJsonSimpleWithJsonSurfer  thrpt   10  155094.948      4457.502  ops/s

Name		Name	Last commit message	Last commit date
Latest commit History 320 Commits
.github		.github
.mvn		.mvn
checkstyle		checkstyle
jsurfer-all		jsurfer-all
jsurfer-benchmark		jsurfer-benchmark
jsurfer-core		jsurfer-core
jsurfer-fastjson		jsurfer-fastjson
jsurfer-gson		jsurfer-gson
jsurfer-jackson-jr		jsurfer-jackson-jr
jsurfer-jackson		jsurfer-jackson
jsurfer-jsonsimple		jsurfer-jsonsimple
.gitignore		.gitignore
.travis.yml		.travis.yml
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
mvnw		mvnw
mvnw.cmd		mvnw.cmd
pom.xml		pom.xml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

SqlJsonSurfer

JsonSurfer - Let's surf on Json!

Why JsonSurfer

Getting started

What is JsonPath?

Usage:

Create your JsonSurfer:

Collect value by JsonPath

"Surfing" in Json and collecting matched value in the listeners

Reuse listener binding.

Compiled JsonPath

JsonPath Filters

Resolver API:

Binaray format (Jackson only)

Share data among processors

Control parsing

Java 8 Streams API support

Non-Blocking parsing

Examples

Find the authors of all books:

All authors

All things in store

The price of everything in the store

The third book

The first two books

Filter all books whose price equals to 8.95

Filter all books which belong to fiction category

All books matching regex

Stoppable parsing

Benchmark

About

Releases

Packages

Languages

License

TomaszGaweda/JsonSurfer

Folders and files

Latest commit

History

Repository files navigation

SqlJsonSurfer

JsonSurfer - Let's surf on Json!

Why JsonSurfer

Getting started

What is JsonPath?

Usage:

Create your JsonSurfer:

Collect value by JsonPath

"Surfing" in Json and collecting matched value in the listeners

Reuse listener binding.

Compiled JsonPath

JsonPath Filters

Resolver API:

Binaray format (Jackson only)

Share data among processors

Control parsing

Java 8 Streams API support

Non-Blocking parsing

Examples

Find the authors of all books:

All authors

All things in store

The price of everything in the store

The third book

The first two books

Filter all books whose price equals to 8.95

Filter all books which belong to fiction category

All books matching regex

Stoppable parsing

Benchmark

About

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages