| layout | title | nav_order | permalink |
|---|---|---|---|
default |
How it works |
3 |
/how-it-works |
CueLake is built upon three core open source products - Zeppelin, Spark SQL, and Iceberg.
You write SQL statements or code as paragraphs in a Zeppelin notebook. When you run a notebook, it's paragraphs run in a sequence. CueLake provides prebuilt notebook templates for common use cases like incremental refresh.
You add one or more notebooks to a Workflow. When you run a workflow, it's notebooks run in parallel.
To create a DAG of workflows, you trigger a workflow after another workflow.
To run a workflow or a notebook as per schedule, you assign it a schedule.
CueLake monitors the execution of workflows and notebooks, and maintains logs for each notebook paragraph.
On each trigger of a schedule, assigned notebooks are queued for execution. CueLake then starts the Spark cluster and runs the queued notebooks. Once notebooks are executed, the Spark cluster is automatically shut down.