-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make ingestion model optional in update_db_model function #67
Labels
Projects
Comments
5 tasks
Closing as this is no longer necessary as the update code will be removed in #74. Using deterministic keys lets us just upsert so the |
Closing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
upload_db_model
, we allowingestion_model
to be an optional param. However, if the db model already exists and an ingestion model isn't passed, then a uniqueness error is thrown.upload_db_model_task
without a corresponding ingestion model because we don't really need an ingestion model. Thfile
db models are not. This could be happen if we're doing something like data cleanup or a backfill which causes inconsistencies.upload_db_model
logic and removing the ingestion modelis not None
check, and changingupdate_db_model
to take inUnion[Model, IngestionModel]
instead. It may also make more sense to have separate params for a db model vs ingestion model inupdate_db_model
since the logic may differ a bitupload_db_model
to only accept adb_model
instead ofingestion_model
, andupdate_db_model
to take inexisting_db_model
andproposed_db_model
. This would probably make the both functions more simple, however would require us to create extra db models outside of those functionsExpected Behavior
upload_db_model_task
is called withNone
for ingestion_model when an existing db model is presentThe text was updated successfully, but these errors were encountered: