Table of Contents
	qsv is a command line program for indexing, slicing, analyzing, filtering, enriching, validating & joining CSV files. Commands are simple, fast & composable. * Available Commands * Installation * Whirlwind Tour * Cookbook * FAQ * Changelog * Performance Tuning * Benchmarks * NYC School of Data 2022 slides * Sponsor

ℹ️ NOTE: qsv is a fork of the popular xsv utility, merging several pending PRs since xsv 0.13.0's May 2018 release. It also has numerous new features & 61 additional commands/subcommands/operations (for a total of 81). See FAQ for more details.

Pre-built binaries for Windows, Linux and macOS are available for download, including binaries compiled with Rust Nightly/Unstable (more info).

There are four variants of qsv:

• qsv enables all features valid for the target platform⁶
• qsvnp enables all features EXCEPT python ("np" stands for "no python")
• qsvlite has all features disabled (~half the size of qsv)
• qsvdp is optimized for use with DataPusher+, with only DataPusher+ relevant commands and the self-update engine removed (~sixth of the size of qsv).

Alternatively, you can install from source by installing Rust and installing qsv using Rust's cargo command⁷:

cargo install qsv --features all_full

If you encounter compilation errors, ensure you have the Python development libraries installed and you're using the exact version of the dependencies qsv was built with by issuing:

cargo install qsv --locked --features all_full

The binary will be installed in ~/.cargo/bin.

Compiling from source also works similarly:

git clone [email protected]:jqnatividad/qsv.git
cd qsv
cargo build --release --features all_full
# or if you encounter compilation errors
cargo build --release --locked --features all_full

The compiled binary will end up in ./target/release/.

To enable optional features, use cargo --features (see Feature Flags for more info):

cargo install qsv --features apply,generate,lua,fetch,foreach,python,self_update,full
# or shorthand
cargo install qsv --features all_full
# or to install all features EXCEPT python
cargo install qsv --features nopython_full
# or to install qsvlite
cargo install qsv --features lite
# or to install qsvdp
cargo install qsv --features datapusher_plus

# or when compiling from a local repo
cargo build --release --features apply,generate,lua,fetch,foreach,python,self_update,full
# shorthand
cargo build --release --features all_full
# all features EXCEPT python
cargo build --release --features nopython_full
# for qsvlite
cargo build --release --features lite
# for qsvdp
cargo build --release --features datapusher_plus

qsv's MSRV policy is to require Rust stable - currently version 1.64.

qsv's command-line options are quite extensive. Thankfully, since it uses docopt for CLI processing, we can take advantage of docopt.rs' tab completion support to make it easier to use qsv at the command-line (currently, only bash shell is supported):

# install docopt-wordlist
cargo install docopt

# IMPORTANT: run these commands from the root directory of your qsv git repository
# to setup bash qsv tab completion
echo "DOCOPT_WORDLIST_BIN=\"$(which docopt-wordlist)"\" >> $HOME/.bash_completion
echo "source \"$(pwd)/scripts/docopt-wordlist.bash\"" >> $HOME/.bash_completion
echo "complete -F _docopt_wordlist_commands qsv" >> $HOME/.bash_completion

qsv recognizes UTF-8/ASCII encoded, CSV (.csv) and TSV files (.tsv and .tab). CSV files are assumed to have "," (comma) as a delimiter, and TSV files, "\t" (tab) as a delimiter. The delimiter is a single ascii character that can be set either by the --delimiter command-line option or with the QSV_DEFAULT_DELIMITER environment variable or automatically detected when QSV_SNIFF_DELIMITER is set.

When using the --output option, note that qsv will UTF-8 encode the file and automatically change the delimiter used in the generated file based on the file extension - i.e. comma for .csv, tab for .tsv and .tab files.

JSONL/NDJSON files are also recognized and converted from/to CSV with the jsonl and tojsonl commands respectively.

The fetch & fetchpost commands also produces JSONL files when its invoked without the --new-column option, and TSV files with the --report option.

The sniff, sortcheck and validate commands produce JSON files with their --json and --pretty-json options.

The schema command produces a JSON Schema Validation (Draft 7) file with the ".schema.json" file extension, which can be used with the validate command.

The excel command recognizes Excel and Open Document Spreadsheet(ODS) files (.xls, .xlsx, .xlsm, .xlsb and .ods files).

qsv validates against the RFC 4180 CSV standard. However IRL, CSV formats vary significantly and qsv is actually not strictly compliant with the specification so it can process "real-world" CSV files. qsv leverages the awesome Rust CSV crate to read/write CSV files.

Click here to find out more about how qsv conforms to the standard using this crate.

The following commands require UTF-8 encoded input (of which ASCII is a subset) - dedup, exclude, fetch, fetchpost, frequency, join, schema, sort, stats & validate.

For these commands, qsv checks if the input is UTF-8 encoded by scanning the first 8k, and will abort if its not unless QSV_SKIPUTF8_CHECK is set. On Linux and macOS, UTF-8 encoding is the default.

This was done to increase performance of these commands, as they make extensive use of from_utf8_unchecked so as not to pay the repetitive utf-8 validation penalty, no matter how small, even for already utf-8 encoded files.

Should you need to re-encode CSV/TSV files, you can use the input command to transcode to UTF-8. It will replace all invalid UTF-8 sequences with �. Alternatively, there are several utilities you can use to do so on Linux/macOS and Windows.

Unlike other modern operating systems, Microsoft Windows' default encoding is UTF16-LE. This will cause problems when redirecting qsv's output to a CSV file and trying to open it with Excel (which ignores the comma delimiter, with everything in the first column):

qsv stats wcp.csv > wcpstats.csv

Which is weird, since you would think Microsoft's own Excel would properly recognize UTF16-LE encoded CSV files. Regardless, to create a properly UTF-8 encoded file, use the --output option instead:

# so instead of redirecting stdout to a file
qsv stats wcp.csv > wcpstats.csv

# do this instead
qsv stats wcp.csv --output wcpstats.csv

With the python feature, qsv will look for Python shared libraries (libpython* on Linux/macOS, python*.dll on Windows) against which it was compiled, and abort with an error if not found, detailing the Python library it was looking for.

Note that this will happen as soon as the qsv binary is invoked, even if you're not running the py command.

If you don't need to run the py command, simply use qsvnp ("np" stands for "no python"), qsvlite, or qsvdp.

If you need the py command, the prebuilt qsv binary is compiled, as a policy, using the current stable Python minor version (currently Python 3.10) at the time of release.

If you require a different Python version (Python 3.6 and up are supported), you'll need to install/compile from source, making sure you have the development libraries for the desired Python version installed when doing so.

PyO3 - the underlying crate that enables the python feature, uses a build script to determine the Python version and set the correct linker arguments. By default it uses the python3 executable. You can override the Python interpreter by setting PYO3_PYTHON (e.g., PYO3_PYTHON=python3.6), before installing/compiling qsv. See the PyO3 User Guide for more information.

If you're distributing python-enabled qsv, you can also "bundle" the Python shared library by including it in the same directory as the qsv binary. qsv will automatically use the "bundled" library instead of the default Python version in the environment.

Also, consider using the lua/luajit commands instead of the py command if the mapping/filtering operation you're trying to do can be done with lua/luajit. Lua is much faster than Python and LuaJIT is even faster still. In addition, Lua/LuaJIT is embedded into qsv and has no external dependencies, unlike Python.

Several dependencies also have environment variables that influence qsv's performance & behavior:

• Memory Management (mimalloc)
  When incorporating qsv into a data pipeline that runs in batch mode, particularly with very large CSV files using qsv commands that load entire CSV files into memory, you can fine-tune Mimalloc's behavior using its environment variables.
• Network Access (reqwest)
  qsv uses reqwest for its fetch, validate and --update functions and will honor proxy settings set through the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables.

ℹ️ NOTE: To get a list of all active qsv-relevant environment variables, run qsv --envlist. Relevant env vars are defined as anything that starts with QSV_ and MIMALLOC_, and the proxy variables listed above.

qsv has several features:

• mimalloc (default) - use the mimalloc allocator (see Memory Allocator for more info).
• apply - enable apply command. This swiss-army knife of CSV transformations is very powerful, but it has a lot of dependencies that increases both compile time and binary size.
• fetch - enables the fetch and fetchpost commands.
• generate - enable generate command.
• full - enable to build qsv binary variant which is feature-capable.
• all_full - enable to build qsv binary variant with all features enabled (apply,fetch,foreach,generate,luajit,python).
• nopython_full - enable to build qsvnp binary variant with all features (apply,fetch,foreach,generate,luajit) EXCEPT python.
• lite - enable to build qsvlite binary variant with all features disabled.
• datapusher_plus - enable to build qsvdp binary variant - the DataPusher+ optimized qsv binary.
• nightly - enable to turn on nightly/unstable features in the rand, regex, hashbrown, parking_lot and pyo3 crates when building with Rust nightly/unstable.
• self_update - enable self-update engine, checking GitHub for the latest release.

The following "power-user" features can be abused and present "foot-shooting" scenarios:

• lua - enable lua command. Embeds a Lua 5.4 interpreter into qsv.
• luajit - enable luajit command. Embeds a LuaJIT 2.0 interpreter into qsv. LuaJIT is a Just-In-Time compiler for the Lua 5.1 language and is thus much faster than Lua. Note that the lua and luajit interpreters are mutually exclusive features.
• foreach - enable foreach command (not valid for Windows).
• python - enable py command (requires Python 3.6+ shared library). Note that qsv will look for the Python shared library (libpython.* on Linux/macOS, python*.dll on Windows) for the Python version it was compiled against and will abort if the library is not found, even if you're not using the py command. Check Python section for more info.

ℹ️ NOTE: qsvlite, as the name implies, always has non-default features disabled. qsv can be built with any combination of the above features using the cargo --features & --no-default-features flags. The pre-built qsv binaries has all applicable features valid for the target platform⁶.

Dual-licensed under MIT or the UNLICENSE.

qsv was made possible by
Standards-based, best-of-breed, open source solutions to make your Data Useful, Usable & Used.

This project is unrelated to Intel's Quick Sync Video.