Guides#

Learn to apply Dagster concepts to your work, explore experimental features, and check out some examples.

Development environment#

• Running Dagster locally - Learn how to run Dagster and its web UI on your local machine

• Using environment variables and secrets in Dagster - Learn to define environment variables and use them to securely use secrets and parameterize your Dagster pipelines

• Transitioning data pipelines from development to production - Learn how to seamlessly transition your Dagster pipelines from local development to production

• Testing against production with Dagster Cloud Branch Deployments - Use Dagster Cloud Branch Deployments to quickly iterate on your Dagster code without impacting production data

Working with data assets#

• Understanding how assets relate to ops and graphs - Learn how software-defined assets relate to ops and graphs, and when to use one over the other

• Moving to Software-defined Assets - Already using ops and graphs, but not Software-defined Assets? Learn why and how to use Software-defined Assets

• Using Software-defined assets with Pandas and PySpark - A quick introduction to Sofware-defined Assets, featuring Pandas and PySpark

• Testing assets - Learn to test your Software-defined Assets

• Migrating to Pythonic resources and config - Incrementally migrate existing Dagster codebases to Pythonic resources and config

Working with tasks#

• Intro to ops and jobs - Learn to execute tasks that don't produce assets

• Re-executing Dagster jobs - Learn to re-execute Dagster jobs using either the UI or Dagster's APIs

• Migrating to graphs, jobs, and ops - Legacy. Migrate to Dagster graphs, jobs, and ops from Dagster solids and pipelines

Migrating to Dagster#

• Migrating from Airflow to Dagster - Perform a lift-and-shift migration from Airflow to Dagster

Best practices#

• Structuring your Dagster project - Learn about Dagster's recommendations on structuring larger projects to help stay organized and efficient

• Automating your data pipelines - Learn how to automate your data pipelines with Dagster using schedules and sensors

• Building machine learning pipelines - Learn about how to use Dagster to build a machine learning pipeline

• Managing machine learning models with Dagster - Review ways to manage and maintain your machine learning (ML) models in Dagster

• Exploring a fully-featured Dagster project - A walkthrough of multiple patterns using a practical, fully-featured Dagster project

Advanced and experimental features#

Advanced features#

• Limiting concurrency in data pipelines - Learn how limiting concurrency in your data pipelines can help prevent performance problems and downtime

• Customizing run queue priority - Define custom prioritization rules for your Dagster instance's run queue

• Validating data with Dagster Type factories - Explore using a Dagster Type factory to validate Pandas DataFrames using Pandera

Experimental features#

• Using Custom Run Coordinators to perform run attribution - A look at using a Custom Run Coordinator to perform run attribution

• Airbyte ingestion as code - Configure Airbyte connections with Dagster

• Asset versioning and caching - Memoize assets using Dagster's data versioning system