forked from dagster-io/dagster
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[docs] - Add testing guide for schedules (DOC-192) (dagster-io#21606)
## Summary & Motivation This PR moves the **Testing** section of the Schedules concept page to its own guide. ## How I Tested These Changes 👀
- Loading branch information
1 parent
a65c013
commit d76d562
Showing
2 changed files
with
187 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,187 @@ | ||
--- | ||
title: "Testing schedules | Dagster Docs" | ||
description: "Learn to test your schedules in Python or the Dagster UI." | ||
--- | ||
|
||
# Testing schedules | ||
|
||
In this guide, we'll show you how to use the Dagster UI and Python to test your [schedules](/concepts/partitions-schedules-sensors/schedules). For more information about testing in Dagster, refer to the [Testing documentation](/concepts/testing). | ||
|
||
--- | ||
|
||
## Testing schedules in the Dagster UI | ||
|
||
Using the UI, you can manually trigger test evaluations of a schedule and view the results. This can be helpful when [creating a schedule](/concepts/automation/schedules/automating-assets-schedules-jobs) or for [troubleshooting unexpected scheduling behavior](/concepts/automation/schedules/troubleshooting). | ||
|
||
1. In the UI, click **Overview > Schedules tab**. | ||
|
||
2. Click the schedule you want to test. | ||
|
||
3. Click the **Test Schedule** button, located near the top right corner of the page. | ||
|
||
4. You'll be prompted to select a mock schedule evaluation time. As schedules are defined on a cadence, the evaluation times in the dropdown are past and future times along that cadence. | ||
|
||
For example, let's say you're testing a schedule with a cadence of `"Every day at X time"`. In the dropdown, you'd see past and future evaluation times along that cadence: | ||
|
||
<!-- ![Selecting a mock evaluation time for a schedule in the Dagster UI](/images/concepts/partitions-schedules-sensors/schedules/testing-select-timestamp-page.png) --> | ||
|
||
<Image | ||
alt="Selecting a mock evaluation time for a schedule in the Dagster UI" | ||
src="/images/concepts/automation/schedules/testing-select-timestamp-page.png" | ||
width={2638} | ||
height={1388} | ||
/> | ||
|
||
5. After selecting an evaluation time, click the **Evaluate** button. | ||
|
||
A window containing the evaluation result will display after the test completes. If the evaluation was successful, click **Open in Launchpad** to launch a run with the same config as the test evaluation. | ||
|
||
--- | ||
|
||
## Testing schedules in Python | ||
|
||
You can also test your schedules directly in Python. In this section, we'll demonstrate how to test: | ||
|
||
- [`@schedule`-decorated functions](#testing-schedule-decorated-functions) | ||
- [Schedules with resources](#testing-schedules-with-resources) | ||
|
||
### Testing @schedule-decorated functions | ||
|
||
To test a function decorated by the <PyObject object="schedule" decorator /> decorator, you can invoke the schedule definition like it's a regular Python function. The invocation will return run config, which can then be validated using the <PyObject object="validate_run_config" /> function. | ||
|
||
Let's say we want to test the `configurable_job_schedule` in this example: | ||
|
||
```python file=concepts/partitions_schedules_sensors/schedules/schedules.py startafter=start_run_config_schedule endbefore=end_run_config_schedule | ||
@op(config_schema={"scheduled_date": str}) | ||
def configurable_op(context: OpExecutionContext): | ||
context.log.info(context.op_config["scheduled_date"]) | ||
|
||
|
||
@job | ||
def configurable_job(): | ||
configurable_op() | ||
|
||
|
||
@schedule(job=configurable_job, cron_schedule="0 0 * * *") | ||
def configurable_job_schedule(context: ScheduleEvaluationContext): | ||
scheduled_date = context.scheduled_execution_time.strftime("%Y-%m-%d") | ||
return RunRequest( | ||
run_key=None, | ||
run_config={ | ||
"ops": {"configurable_op": {"config": {"scheduled_date": scheduled_date}}} | ||
}, | ||
tags={"date": scheduled_date}, | ||
) | ||
``` | ||
|
||
To test this schedule, we used <PyObject object="build_schedule_context" /> to construct a <PyObject object="ScheduleEvaluationContext" /> to provide to the `context` parameter: | ||
|
||
```python file=concepts/partitions_schedules_sensors/schedules/schedule_examples.py startafter=start_test_cron_schedule_context endbefore=end_test_cron_schedule_context | ||
from dagster import build_schedule_context, validate_run_config | ||
|
||
|
||
def test_configurable_job_schedule(): | ||
context = build_schedule_context( | ||
scheduled_execution_time=datetime.datetime(2020, 1, 1) | ||
) | ||
run_request = configurable_job_schedule(context) | ||
assert validate_run_config(configurable_job, run_request.run_config) | ||
``` | ||
|
||
If your <PyObject object="schedule" decorator />-decorated function doesn't have a context parameter, you don't need to provide one when invoking it. | ||
|
||
### Testing schedules with resources | ||
|
||
For schedules that utilize [resources](/concepts/resources), you can provide the resources when invoking the schedule function. | ||
|
||
Let's say we want to test the `process_data_schedule` in this example: | ||
|
||
```python file=/concepts/resources/pythonic_resources.py startafter=start_new_resource_on_schedule endbefore=end_new_resource_on_schedule dedent=4 | ||
from dagster import ( | ||
schedule, | ||
ScheduleEvaluationContext, | ||
ConfigurableResource, | ||
job, | ||
RunRequest, | ||
RunConfig, | ||
Definitions, | ||
) | ||
from datetime import datetime | ||
from typing import List | ||
|
||
class DateFormatter(ConfigurableResource): | ||
format: str | ||
|
||
def strftime(self, dt: datetime) -> str: | ||
return dt.strftime(self.format) | ||
|
||
@job | ||
def process_data(): ... | ||
|
||
@schedule(job=process_data, cron_schedule="* * * * *") | ||
def process_data_schedule( | ||
context: ScheduleEvaluationContext, | ||
date_formatter: DateFormatter, | ||
): | ||
formatted_date = date_formatter.strftime(context.scheduled_execution_time) | ||
|
||
return RunRequest( | ||
run_key=None, | ||
tags={"date": formatted_date}, | ||
) | ||
|
||
defs = Definitions( | ||
jobs=[process_data], | ||
schedules=[process_data_schedule], | ||
resources={"date_formatter": DateFormatter(format="%Y-%m-%d")}, | ||
) | ||
``` | ||
|
||
In the test for this schedule, we provided the `date_formatter` resource to the schedule when we invoked its function: | ||
|
||
```python file=/concepts/resources/pythonic_resources.py startafter=start_test_resource_on_schedule endbefore=end_test_resource_on_schedule dedent=4 | ||
from dagster import build_schedule_context, validate_run_config | ||
|
||
def test_process_data_schedule(): | ||
context = build_schedule_context( | ||
scheduled_execution_time=datetime.datetime(2020, 1, 1) | ||
) | ||
run_request = process_data_schedule( | ||
context, date_formatter=DateFormatter(format="%Y-%m-%d") | ||
) | ||
assert ( | ||
run_request.run_config["ops"]["fetch_data"]["config"]["date"] | ||
== "2020-01-01" | ||
) | ||
``` | ||
|
||
--- | ||
|
||
## APIs in this guide | ||
|
||
| Name | Description | | ||
| ----------------------------------------------- | ------------------------------------------------------------------------------------- | | ||
| <PyObject object="schedule" decorator /> | Decorator that defines a schedule that executes according to a given cron schedule. | | ||
| <PyObject object="validate_run_config" /> | A function that validates a provided run config blob against a job. | | ||
| <PyObject object="build_schedule_context" /> | A function that constructs a `ScheduleEvaluationContext`, typically used for testing. | | ||
| <PyObject object="ScheduleEvaluationContext" /> | The context passed to the schedule definition execution function. | | ||
|
||
--- | ||
|
||
## Related | ||
|
||
<ArticleList> | ||
<ArticleListItem | ||
title="Troubleshooting schedules" | ||
href="/concepts/automation/schedules/troubleshooting" | ||
></ArticleListItem> | ||
<ArticleListItem | ||
title="Automating assets using schedules and jobs" | ||
href="/concepts/automation/schedules/automating-assets-schedules-jobs" | ||
></ArticleListItem> | ||
<ArticleListItem | ||
title="Automating ops using schedules and jobs" | ||
href="/concepts/automation/schedules/automating-ops-schedules-jobs" | ||
></ArticleListItem> | ||
<ArticleListItem title="Testing" href="/concepts/testing"></ArticleListItem> | ||
</ArticleList> |
Binary file added
BIN
+369 KB
...t/public/images/concepts/automation/schedules/testing-select-timestamp-page.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.