-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue upgrading Dagster to 1.7.15 using helm/celery/k8 worker #23268
Comments
Hi @henrytseng - just to confirm, the thing that you upgraded to 1.7.15 is the image running your code, while leaving the rest of the helm chart using the 1.7.14 release? |
Summary: Resolves #23268. #23053 added a new argument to the celery task signature and passed it in, even when it was empty. Since the celery workers might be running older versions of dagster, a new version of dagster running the executor might set this field and hit the error in that task. Attempt to resolve that in two directions: - only set the new argument if it is non-empty - Add kwargs to the task signature, to hopefully make it more resilient to new arguments in the future Test Plan: BK, run celery with user code on master but the celery workers on 1.7.14, should no longer reproduce the linked error
Hi, @henrytseng , I'm also experiencing same issue after upgrading to 1.7.15 in package and keeping docker image 1.7.14. |
Thanks for flagging that - we've identified the problem that occurred there and are pushing the images now. |
The missing 1.7.15 image has now been pushed. |
Dagster version
dagster, version 1.7.15
What's the issue?
I'm seeing an error with after upgrading from
1.7.14
to1.7.15
.Just a heads up, it looks like
per_step_k8s_config
was just added to1.7.15
but it doesn't seem like it works with the earlier1.7.14
image in DockerHub.What did you expect to happen?
Minor version patch compatibility
How to reproduce?
Run
celeryK8sRunLauncher
with Helm chart with 1.7.14 image deployed and kick off Celery based job.Deployment type
Dagster Helm chart
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
The text was updated successfully, but these errors were encountered: