Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] use pip sync instead of pip install for pinned dependencies #87

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

cnolanminich
Copy link
Contributor

@cnolanminich cnolanminich commented Apr 19, 2024

This PR introduces a way to separate the python packages you need vs. the pinned dependencies.

The motivating factors to adopt this:

  • we can control when we upgrade Dagster versions for Hooli (so if there is a breaking change we're not quite ready to adopt, we can accept other PRs in the meantime).
  • We can see all of the python packages that are used (so if there are weird dependency issues we can see them in the requirements.txt, which acts as lockfile of sorts.
  • The Dockerfile builds deterministically (so if it built once it will always build).
  • It's useful to have a demonstration of how to do this in case customers want to see how we would use pinned dependencies
  • (later) we could likely skip the "build steps" if these files aren't updated, which would allow build to be much faster (< 1 minute vs. 8 minutes)

Why we might not want to

  • it's another step if you want to upgrade anything in Dagster+
  • you need Docker installed on your computer because uv pip sync is platform-specific (this is fine for every package except for Sling, which requires amd64 arch to be specified.
  • it's annoying if your setup.py would work across Python versions but are set to different versions (as we have in some of our Dockerfiles). The only way to solve this permanently would be multiple dockerfiles with all of our python distros.

Copy link

Your pull request at commit d55b0dd838da0fe8722f3fab84671522158a3785 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Building... Apr 19, 2024 at 02:25 PM (UTC)
batch_enrichment Building... Apr 19, 2024 at 02:25 PM (UTC)
data-eng-pipeline Building... Apr 19, 2024 at 02:25 PM (UTC)
basics Building... Apr 19, 2024 at 02:25 PM (UTC)
snowflake_insights Building... Apr 19, 2024 at 02:25 PM (UTC)

Copy link

github-actions bot commented Apr 19, 2024

Your pull request at commit 734f8d1e15a213f92d88164bfa30a14a6ba6929f is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Deploy failed Apr 19, 2024 at 04:29 PM (UTC)
batch_enrichment Deploy failed Apr 19, 2024 at 04:29 PM (UTC)
data-eng-pipeline Deploy failed Apr 19, 2024 at 04:29 PM (UTC)
basics Deploy failed Apr 19, 2024 at 04:29 PM (UTC)
snowflake_insights Deploy failed Apr 19, 2024 at 04:29 PM (UTC)

Copy link

github-actions bot commented Apr 19, 2024

Your pull request at commit a1b5eacfca0ec05832322f499af095d8c412fdd9 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Deploy failed Apr 19, 2024 at 05:23 PM (UTC)
batch_enrichment Deploy failed Apr 19, 2024 at 05:23 PM (UTC)
data-eng-pipeline Deploy failed Apr 19, 2024 at 05:23 PM (UTC)
basics Deploy failed Apr 19, 2024 at 05:23 PM (UTC)
snowflake_insights Deploy failed Apr 19, 2024 at 05:23 PM (UTC)

Copy link

github-actions bot commented Apr 19, 2024

Your pull request at commit 342356224a519bd61a918b6b3a671130fb162417 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets Deploy failed Apr 19, 2024 at 05:36 PM (UTC)
batch_enrichment Deploy failed Apr 19, 2024 at 05:36 PM (UTC)
data-eng-pipeline Deploy failed Apr 19, 2024 at 05:36 PM (UTC)
basics Deploy failed Apr 19, 2024 at 05:36 PM (UTC)
snowflake_insights Deploy failed Apr 19, 2024 at 05:36 PM (UTC)

Copy link

github-actions bot commented Apr 19, 2024

Your pull request at commit ecdf90fd0558d85f3437c835644d21043228f218 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets View in Cloud Apr 19, 2024 at 05:49 PM (UTC)
batch_enrichment View in Cloud Apr 19, 2024 at 05:49 PM (UTC)
data-eng-pipeline View in Cloud Apr 19, 2024 at 05:49 PM (UTC)
basics View in Cloud Apr 19, 2024 at 05:49 PM (UTC)
snowflake_insights View in Cloud Apr 19, 2024 at 05:49 PM (UTC)

Copy link

github-actions bot commented Apr 19, 2024

Your pull request at commit 612609ba387a7409b30b70439cb31de7e46fc087 is automatically being deployed to Dagster Cloud.

Location Status Link Updated
demo_assets View in Cloud Apr 19, 2024 at 06:01 PM (UTC)
batch_enrichment View in Cloud Apr 19, 2024 at 06:01 PM (UTC)
data-eng-pipeline View in Cloud Apr 19, 2024 at 06:01 PM (UTC)
basics View in Cloud Apr 19, 2024 at 06:01 PM (UTC)
snowflake_insights View in Cloud Apr 19, 2024 at 06:01 PM (UTC)

@cnolanminich cnolanminich changed the title use pip sync instead of pip install [RFC] use pip sync instead of pip install for pinned dependencies Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant