Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make ingestion model optional in update_db_model function #67

Closed
isaacna opened this issue Jul 6, 2021 · 2 comments · Fixed by #69
Closed

Make ingestion model optional in update_db_model function #67

isaacna opened this issue Jul 6, 2021 · 2 comments · Fixed by #69
Labels
bug Something isn't working enhancement New feature or request
Projects

Comments

@isaacna
Copy link
Collaborator

isaacna commented Jul 6, 2021

Description

  • In upload_db_model, we allow ingestion_model to be an optional param. However, if the db model already exists and an ingestion model isn't passed, then a uniqueness error is thrown.
  • There are some places in the event gather pipeline where we call upload_db_model_task without a corresponding ingestion model because we don't really need an ingestion model. Th
  • This is an unlikely edge case, but there could be a situation where an audio file is deleted from the file store, but the corresponding file db models are not. This could be happen if we're doing something like data cleanup or a backfill which causes inconsistencies.
  • There are a few different ways to refactor this
    • We could fix this by changing the upload_db_model logic and removing the ingestion model is not None check, and changing update_db_model to take in Union[Model, IngestionModel] instead. It may also make more sense to have separate params for a db model vs ingestion model in update_db_model since the logic may differ a bit
    • We could also change upload_db_model to only accept a db_model instead of ingestion_model, and update_db_model to take in existing_db_model and proposed_db_model. This would probably make the both functions more simple, however would require us to create extra db models outside of those functions

Expected Behavior

  • Uniqueness error to not get triggered when upload_db_model_task is called with None for ingestion_model when an existing db model is present
@evamaxfield evamaxfield added bug Something isn't working enhancement New feature or request labels Jul 7, 2021
@sarahjliu sarahjliu added this to To do in v3.0 Jul 10, 2021
@sarahjliu sarahjliu moved this from Ready for Dev to Done in v3.0 Jul 10, 2021
@sarahjliu sarahjliu moved this from Done to Ready for Dev in v3.0 Jul 10, 2021
v3.0 automation moved this from Ready for Dev to Done Jul 25, 2021
@isaacna isaacna reopened this Jul 27, 2021
v3.0 automation moved this from Done to In progress Jul 27, 2021
@isaacna
Copy link
Collaborator Author

isaacna commented Aug 9, 2021

Closing as this is no longer necessary as the update code will be removed in #74.

Using deterministic keys lets us just upsert so the update_db_model function will be removed.

@isaacna
Copy link
Collaborator Author

isaacna commented Aug 9, 2021

Closing

@isaacna isaacna closed this as completed Aug 9, 2021
v3.0 automation moved this from In progress to Done Aug 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
No open projects
v3.0
Done
Development

Successfully merging a pull request may close this issue.

2 participants