Dagster is an opinionated system and programming model for data pipelines. This process goes by many names -- ETL (extract-transform-load), ELT (extract-load-transform), model production, data integration, and so on -- but in essence they all describe the same activity: Performing a set of computations structured as a DAG (directed, acyclic graph) that end up producing data assets, whether those assets be tables, files, machine-learning models, etc.

There are a few tools in this repo:

• Dagster: The core programming model and abstraction stack; a stateless single-node and -process execution engine; and a CLI tool for driving that engine.

• Dagit: Dagit is a rich viewer for Dagster assets.
• Dagster GE: A Dagster integration with Great Expectations. (see https://github.com/great-expectations/great_expectations)
• Dagstermill: An experimental prototype for integrating productionized notebooks into dagster pipelines. Built on the papermill library (https://github.com/nteract/papermill)

Go to https://dagster.readthedocs.io/en/latest/ for documentation!