Dagster Integration:

About this integration

SDF can integrate seemlessly with your existing Dagster projects, providing the best-in-class transformation layer while enabling you to schedule, orchestrate, and monitor your dags in Dagster.

When it comes time to materialize your Dagster assets, you can be confident that SDF has successfully compiled your workspace, making it safe to execute locally or against your cloud data warehouse.

Installation

pip install dagster-sdf

Example

from pathlib import Path

import dagster as dg
from dagster_sdf import (
    SdfCliResource,
    SdfWorkspace,
    sdf_assets,
)

workspace_dir = Path(__file__).joinpath("./my_sdf_workspace").resolve()
target_dir = workspace_dir.joinpath(
    "sdf_dagster_out"
)  # The destination for outputs generated by SDF during execution
environment = "dbg"  # Replace with your environment, e.g. "prod"

workspace = SdfWorkspace(
    workspace_dir=workspace_dir,
    target_dir=target_dir,
    environment=environment,
)


@sdf_assets(workspace=workspace)
def my_sdf_assets(context: dg.AssetExecutionContext, sdf: SdfCliResource):
    yield from sdf.cli(
        ["run", "--save", "info-schema"],
        target_dir=target_dir,
        environment=environment,
        context=context,
    ).stream()


defs = dg.Definitions(
    assets=[my_sdf_assets],
    resources={
        "sdf": SdfCliResource(workspace_dir=workspace_dir),
    },
)

About SDF

SDF is a multi-dialect SQL compiler, transformation framework, and analytical database engine. It natively compiles SQL dialects, like Snowflake, and connects to their corresponding data warehouses to materialize models.