Skip to content

Command line (CLI) tool to inspect Apache Parquet files on the go

License

Notifications You must be signed in to change notification settings

chhantyal/parquet-cli

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

parquet-cli

Command line (CLI) tool to inspect Apache Parquet files on the go

Apache Parquet is a columnar storage format commonly used in the Hadoop ecosystem.

parq is small, easy to install, Python utility to view and get basic information from Parquet files.

Current features set are what I need, please use Github issues for any requests/suggestions.

Install

pip install parquet-cli

An executable script called parq will be installed.

Use

Once installed, you can use parq command.

View Parquet file metadata:

$ parq input.parquet

# Metadata
<pyarrow._parquet.FileMetaData object at 0x1014879a8>
created_by: parquet-mr version 1.8.1 (build 4aba4dae7bb0d4edbcf7923ae1339f28fd3f7fcf)
num_columns: 13
num_rows: 1000
num_row_groups: 1
format_version: 1.0
serialized_size: 1125

Get schema information:

$ parq input.parquet --schema

# Schema
<pyarrow._parquet.ParquetSchema object at 0x1048b9a88>
registration_dttm: INT96
id: INT32
name: BYTE_ARRAY UTF8
email: BYTE_ARRAY UTF8
...
ip_address: BYTE_ARRAY UTF8
country: BYTE_ARRAY UTF8

Get total rows count:

$ parq input.parquet --count

1025

Get top N records (head)

$ parq input.parquet --head 10

Get bottom N records (tail)

$ parq input.parquet --tail 10

Help

$ parq --help

usage: usage: parq file [-s [SCHEMA] | --head [HEAD] | --tail [TAIL] | -c [COUNT]]

positional arguments:
  file                  Parquet file

optional arguments:
  -h, --help            show this help message and exit
  -s [SCHEMA], --schema [SCHEMA]
                        get schema information
  --head [HEAD]         get first N rows from file
  --tail [TAIL]         get last N rows from file
  -c [COUNT], --count [COUNT]
                        get total rows count

About

Command line (CLI) tool to inspect Apache Parquet files on the go

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages