Skip to content

Commit

Permalink
Add pretty printer, don't panic when unable to discover shape of colu…
Browse files Browse the repository at this point in the history 
…mn, support concatenated JSON (multiprocessio#17)

* Add pretty printing notes to readme

* Bump to datastation main

* Update readme

* Fix for format

* Update readme
  • Loading branch information
@eatonphil
eatonphil authored Feb 3, 2022
1 parent e8f58be commit 9809db5
Show file tree
Hide file tree
Showing 4 changed files with 390 additions and 30 deletions.
72 changes: 53 additions & 19 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,39 @@ Or:
$ dsq testdata.ndjson "SELECT name, AVG(time) FROM {} GROUP BY name ORDER BY AVG(time) DESC"
```

### Pretty print

By default `dsq` prints ugly JSON. This is the most efficient mode.

```bash
$ dsq testdata/userdata.parquet 'select count(*) from {}'
[{"count(*)":1000}
]
```

If you want prettier JSON you can pipe `dsq` to `jq`.

```bash
$ dsq testdata/userdata.parquet 'select count(*) from {}' | jq
[
{
"count(*)": 1000
}
]
```

Or you can enable pretty printing with `-p` or `--pretty` in `jsq`
which will display your results in an ASCII table.

```bash
$ ./dsq --pretty testdata/userdata.parquet 'select count(*) from {}'
+----------+
| count(*) |
+----------+
| 1000 |
+----------+
```

### Piping data to dsq

When piping data to `dsq` you need to set the `-s` flag and specify
Expand Down Expand Up @@ -165,18 +198,19 @@ Because `location` is not a scalar value. It is an object.

## Supported Data Types

| Name | File Extension(s) | Notes |
|-----------|-|---------------------|
| CSV | `csv` ||
| TSV | `tsv`, `tab` ||
| JSON | `json` | Must be an array of objects. |
| Newline-delimited JSON | `ndjson`, `jsonl` ||
| Parquet | `parquet` ||
| Excel | `xlsx`, `xls` | Currently only works if there is only one sheet. |
| ODS | `ods` | Currently only works if there is only one sheet. |
| Apache Error Logs | `text/apache2error` | Currently only works if being piped in. |
| Apache Access Logs | `text/apache2access` | Currently only works if being piped in. |
| Nginx Access Logs | `text/nginxaccess` | Currently only works if being piped in. |
| Name | File Extension(s) | Mime Type | Notes |
|-----------|-|-|--------------------|
| CSV | `csv` | `text/csv` | |
| TSV | `tsv`, `tab` | `text/tab-separated-values` | |
| JSON | `json` | `application/json` | Must be an array of objects. |
| Newline-delimited JSON | `ndjson`, `jsonl` | `application/jsonlines` | |
| Concatenated JSON | `cjson` | `application/jsonconcat` ||
| Parquet | `parquet` | `parquet` ||
| Excel | `xlsx`, `xls` | `application/vnd.ms-excel` | Currently only works if there is only one sheet. |
| ODS | `ods` |`application/vnd.oasis.opendocument.spreadsheet` | Currently only works if there is only one sheet. |
| Apache Error Logs | NA | `text/apache2error` | Currently only works if being piped in. |
| Apache Access Logs | NA | `text/apache2access` | Currently only works if being piped in. |
| Nginx Access Logs | NA | `text/nginxaccess` | Currently only works if being piped in. |

## Engine

Expand All @@ -189,22 +223,22 @@ kinds of SQL queries on arbitrary (structured) data.

| Name | Link | Supported File Types | Engine |
|----|-|-|------------------------------------------------------------------------|
| q | https://harelba.github.io/q/ | CSV, TSV | Uses SQLite |
| textql | https://github.com/dinedal/textql | CSV, TSV | Uses SQLite |
| octoql | https://github.com/cube2222/octosql | JSON, CSV, Excel, Parquet | Custom engine missing many features from SQLite |
| dsq | Here | CSV, TSV, JSON, Newline-delimited JSON, Parquet, Excel, ODS (OpenOffice Calc), Logs | Uses SQLite |
| q | https://harelba.github.io/q/ | CSV, TSV | SQLite |
| textql | https://github.com/dinedal/textql | CSV, TSV | SQLite |
| octoql | https://github.com/cube2222/octosql | JSON, CSV, Excel, Parquet | Custom engine |
| dsq | Here | CSV, TSV, a few variations of JSON, Parquet, Excel, ODS (OpenOffice Calc), Logs | SQLite |

## Community

[Join us on Discord](https://discord.gg/f2wQBc4bXX).
[Join us on Discord](https://discord.multiprocess.io).

## How can I help?

Download the app and use it! Report bugs on
[Discord](https://discord.gg/f2wQBc4bXX).
[Discord](https://discord.multiprocess.io).

Before starting on any new feature though, check in on
[Discord](https://discord.gg/f2wQBc4bXX)!
[Discord](https://discord.multiprocess.io)!

## Subscribe

Expand Down
43 changes: 41 additions & 2 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,13 @@ go 1.17

require (
github.com/google/uuid v1.3.0
github.com/multiprocessio/datastation/runner v0.0.0-20220121201025-e665cd7ac0fc
github.com/multiprocessio/datastation/runner v0.0.0-20220203023253-565c744a1e81
github.com/olekukonko/tablewriter v0.0.5
)

require (
cloud.google.com/go v0.65.0 // indirect
cloud.google.com/go/bigquery v1.8.0 // indirect
github.com/Azure/azure-pipeline-go v0.2.3 // indirect
github.com/Azure/azure-storage-blob-go v0.14.0 // indirect
github.com/ClickHouse/clickhouse-go v1.5.1 // indirect
Expand All @@ -24,28 +27,52 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.10.0 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.23.0 // indirect
github.com/aws/smithy-go v1.9.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58 // indirect
github.com/deepmap/oapi-codegen v1.8.2 // indirect
github.com/denisenkom/go-mssqldb v0.11.0 // indirect
github.com/elastic/go-elasticsearch/v6 v6.8.10 // indirect
github.com/fatih/color v1.13.0 // indirect
github.com/form3tech-oss/jwt-go v3.2.5+incompatible // indirect
github.com/gabriel-vasile/mimetype v1.4.0 // indirect
github.com/go-sql-driver/mysql v1.6.0 // indirect
github.com/goccy/go-yaml v1.9.4 // indirect
github.com/gocql/gocql v0.0.0-20211222173705-d73e6b1002a7 // indirect
github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe // indirect
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/flatbuffers v2.0.5+incompatible // indirect
github.com/googleapis/gax-go/v2 v2.0.5 // indirect
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed // indirect
github.com/influxdata/influxdb-client-go/v2 v2.7.0 // indirect
github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/jmoiron/sqlx v1.3.4 // indirect
github.com/jpillora/backoff v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/jstemmer/go-junit-report v0.9.1 // indirect
github.com/klauspost/compress v1.13.6 // indirect
github.com/lib/pq v1.10.4 // indirect
github.com/mattn/go-colorable v0.1.12 // indirect
github.com/mattn/go-ieproxy v0.0.1 // indirect
github.com/mattn/go-isatty v0.0.14 // indirect
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/mattn/go-sqlite3 v1.14.10 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/multiprocessio/go-openoffice v0.0.0-20220110232726-064f5dda1956 // indirect
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
github.com/pierrec/lz4/v4 v4.1.12 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.12.0 // indirect
github.com/prometheus/client_model v0.2.0 // indirect
github.com/prometheus/common v0.32.1 // indirect
github.com/prometheus/procfs v0.7.3 // indirect
github.com/richardlehane/mscfb v1.0.3 // indirect
github.com/richardlehane/msoleps v1.0.1 // indirect
github.com/sijms/go-ora/v2 v2.2.23-0.20220121124241-08ee0a8e5243 // indirect
Expand All @@ -55,9 +82,21 @@ require (
github.com/xitongsys/parquet-go-source v0.0.0-20211228015320-b4f792c43cd0 // indirect
github.com/xuri/efp v0.0.0-20210322160811-ab561f5b45e3 // indirect
github.com/xuri/excelize/v2 v2.5.0 // indirect
go.opencensus.io v0.22.4 // indirect
golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3 // indirect
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 // indirect
golang.org/x/mod v0.4.2 // indirect
golang.org/x/net v0.0.0-20220107192237-5cfca573fb4d // indirect
golang.org/x/sys v0.0.0-20220110181412-a018aaa089fe // indirect
golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c // indirect
golang.org/x/sys v0.0.0-20220114195835-da31bd327af9 // indirect
golang.org/x/text v0.3.7 // indirect
golang.org/x/tools v0.1.4 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/api v0.30.0 // indirect
google.golang.org/appengine v1.6.6 // indirect
google.golang.org/genproto v0.0.0-20210630183607-d20f26d13c79 // indirect
google.golang.org/grpc v1.39.0 // indirect
google.golang.org/protobuf v1.27.1 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
)
Loading

0 comments on commit 9809db5

Please sign in to comment.