qsv is a command line program for indexing, slicing, analyzing, splitting, enriching, validating & joining CSV files. Commands are simple, fast and composable:

1. Simple tasks are easy.
2. Performance trade offs are exposed in the CLI interface.
3. Composition does not come at the expense of performance.

• Available Commands
• Installation
• Whirlwind Tour
• Cookbook
• FAQ
• Benchmarks
• Sponsor

NOTE: qsv is a fork of the popular xsv utility, merging several pending PRs since xsv 0.13.0's release, along with additional features & commands for data-wrangling. See FAQ for more details. (NEW and EXTENDED commands are marked accordingly).

Binaries for Windows, Linux and macOS are available from Github.

Alternatively, you can compile from source by installing Cargo (Rust's package manager) and installing qsv using Cargo:

cargo install qsv

Compiling from this repository also works similarly:

git clone git:https://github.com/jqnatividad/qsv
cd qsv
cargo build --release

The compiled binary will end up in ./target/release/qsv.

Modern CPUs have various features that the Rust compiler can take advantage of to increase performance. If you want the compiler to take advantage of these CPU-specific speed-ups, set this environment variable BEFORE installing/compiling qsv:

On Linux and macOS:

export CARGO_BUILD_RUSTFLAGS='-C target-cpu=native'

On Windows Powershell:

$env:CARGO_BUILD_RUSTFLAGS='-C target-cpu=native'

Do note though that the resulting binary will only run on machines with the same architecture as the machine you installed/compiled from.
To find out your CPU architecture and other valid values for target-cpu:

rustc --print target-cpus

By default, qsv uses an alternative allocator - mimalloc, a performance-oriented allocator from Microsoft. If you want to use the standard allocator, use the --no-default-features flag when installing/compiling qsv, e.g.:

cargo install qsv --no-default-features

or

cargo build --release --no-default-features

Depending on your filesystem's configuration (e.g. block size, SSD, file system type, etc.), you can also fine-tune qsv's read/write buffers.

By default, the read buffer size is set to 16k, you can change it by setting the environment variable QSV_RDR_BUFFER_CAPACITY in bytes.

The same is true with the write buffer (default: 32k) with the QSV_WTR_BUFFER_CAPACITY environment variable.

Use and fine-tune the benchmark script when tweaking qsv's performance to your environment. Don't be afraid to change the benchmark data and the qsv commands to something that is more representative of your workloads.

Use the generated TSV files to meter and compare performance across platforms. You'd be surprised how performance varies across environments - e.g. qsv's join and scramble operations perform abysmally on Windows's WSL running Ubuntu, with join taking 172.44 seconds and scramble, 237.46 seconds. On the same machine, running in a VirtualBox VM at that with the same Ubuntu version, join takes 1.34 seconds, and scramble 2.14 seconds - two orders of magnitude faster!

However, stats performs two times faster on WSL vs the VirtualBox VM - 2.80 seconds vs 5.33 seconds for the stats_index benchmark.

Dual-licensed under MIT or the UNLICENSE.

qsv was made possible by datHere - Data Infrastructure Engineering.
Standards-based, best-of-breed, open source solutions to make your Data Useful, Usable & Used.

This project is unrelated to Intel's Quick Sync Video.