Asset checks#
Using asset checks, you can define and execute different types of checks on your data assets directly in Dagster. Each asset check tests some property of a data asset, such as:
- Ensuring a particular column, like an
ID, doesn't contain null values - Verifying that a tabular asset adheres to a specified schema
- If the asset's data is in need of a refresh
Assets, their checks, and the results of those checks can be viewed in the Dagster UI, providing you with a unified view of your pipeline's health. For example:
Benefits#
Using asset checks helps you:
- Create a single place in the Dagstser UI to learn about and investigate issues in your data pipeline
- Ensure that data produced by your pipeline is timely and high quality by applying basic suites of tests to all assets
- Identify problems in the source data that your pipeline depends on, such as checking data quality on source assets where dbt tests aren't possible
- Communicate what the data produced by your pipeline is expected to look like (aka act as a data contract)
- Raise warnings or block downstream assets from materializing when issues arise
- Track asset check results in the UI, allowing you to monitor data quality metrics over time
Prerequisites#
Before continuing, you should be familiar with:
How it works#
Defined in code, asset checks are used to test some property of one or more Dagster assets. Asset checks can be defined by:
- Using custom Python code
- Imported from external systems like dbt
- Using Dagster's built-in utility methods
Asset checks and their results are visible in the UI, allowing you to communicate useful information about data quality, data freshness, and other issues to stakeholders. Asset check results can also be used to create conditional steps in your pipelines - for example, if a quality check fails, execution can be halted to prevent issues spreading downstream.
Using schedules and sensors, you can automate the execution of jobs that include asset checks and the assets that they target. Checks can also be executed on a one-off basis using the Dagster UI. Refer to the Executing checks section of the Defining and executing asset checks guide for more info.
Getting started#
Check out these guides to get started with asset checks:
From here, you can:
- Block downstream execution based on check results
- Use severity levels and metadata to communicate actionable details about check results in the UI
- Use freshness checks to identify stale assets that are due for a refresh
- Dagster+ only:
- Create alerts based on asset check results
- Use Insights' asset materialization metadata to track asset check results over time
Built-in asset checks#
The following table lists Dagster's built-in utility methods for creating asset checks.
| API | Description |
|---|---|
build_metadata_bounds_checks | Builds asset checks that pass if numeric metadata value falls within a particular range |
build_column_schema_change_checks | Builds asset checks that pass if an asset's columns are the same, compared with its prior materialization |
build_last_update_freshness_checks | Builds asset checks that pass if not too much time has elapsed since the latest time an asset was updated |
build_time_partition_freshness_checks | Builds asset checks that pass if an asset's most recent partition has been materialized before a deadline |
Limitations#
Dagster's UI is tested with a maximum of 1,000 checks per asset. It's designed with the expectation that most assets will have fewer than 50 checks. If you have a use case that doesn't fit these limits, reach out to Dagster support to discuss.
Checks are currently only supported per-asset, not per-partition. See this issue for updates.