Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Customize UserDefinedDagsterK8sConfig per op via tags (dagster-io#23053)
## Summary & Motivation As stated in dagster-io#22138 , Dagster currently requires that op/job limits in k8s are defined statically - and no dynamic overload is allowed. I'd like to resolve this issue for `celery-k8s` launcher (we use them in our production). This PR tries to address the problem by introducing a special tag `"dagster-k8s/config-per-op"` that allows to override `user_defined_k8s_config`. I expect this to work as follows: ```python from dagster import op, job, schedule, RunRequest @op def op1(): print(1 + 2) # op defines it's own limits @op( tags={ "dagster-k8s/config": { "container_config": { "resources": { "requests": {"cpu": "1", "memory": "1Gi"}, "limits": {"cpu": "10", "memory": "10Gi"}, } } } } ) def op2(): print(7) # <------- one way to override op tags @job( tags={ "dagster-k8s/config-per-op": { "op2": { "container_config": { # requests will be overwritten, limits will be taken from op definition "resources": { "requests": {"cpu": "2", "memory": "2Gi"}, } } }, "op1": { "container_config": { # op1 gets requests/limits even though it never defined them "resources": { "requests": {"cpu": "11", "memory": "11Gi"}, "limits": {"cpu": "22", "memory": "22Gi"}, } } }, } } ) def main_job(): op1() op2() # <------- another way to override op tags @schedule(job=main_job, cron_schedule="* * * * *") def main_job(): return RunRequest( tags={ "dagster-k8s/config-per-op": { "op2": { "container_config": { # requests will be overwritten at schedule level, limits will be taken from op definition "resources": { "requests": {"cpu": "8", "memory": "8Gi"}, } } }, "op1": { "container_config": { # op1 gets requests/limits from schedule definition "resources": { "requests": {"cpu": "88", "memory": "88Gi"}, "limits": {"cpu": "89", "memory": "89Gi"}, } } }, } } ) ``` I could use this feature for jobs that vary in resource consumption (for instance, jobs that accept sql query and download data from db). Could you please take a look at the code and check it for broken corner cases or violated contracts? This is my first dive into Dagster codebase so I'll appreciate any comments and suggestions. ## How I Tested These Changes I modified some unit-tests in k8s-related modules in dagster. No integration testing was done. --------- Co-authored-by: gibsondan <[email protected]>
- Loading branch information