Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Clean up stale/unmanaged models in the target schema #278

Open
louisjoecodes opened this issue May 23, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@louisjoecodes
Copy link
Contributor

Problem to be solved

When a user develops models in their Quary project, builds the project (quary build) to materialize those models in the target schema/database, but later deletes some of those model definitions from the project, the corresponding objects remain in the target schema even though they are no longer part of the current project. Over time, this can lead to the accumulation of stale, unused objects in the target schema, increasing cloud storage costs and cluttering the data warehouse.

Quary currently lacks an easy way for users to clean up these orphaned objects and keep their target schema in sync with the current state of their project.

Suggested solution

Implement a new Quary operation, tentatively called quary clean, that will:

  1. Introspect the current Quary project to determine what models/objects it defines
  2. Compare this to the objects actually present in the target schema
  3. Identify any objects in the target that no longer map to any models in the current project
  4. Provide the user clear output listing out these stale objects
  5. Optionally, allow the user to specify a flag that will generate the DDL statements to drop those stale objects
  6. Optionally, allow the user to specify an additional flag to actually execute those drop statements

This will allow users to manually trigger clean-up of their target schemas and remove any models that are no longer part of their current project.

As a potential incremental step towards this goal, we could first implement a simpler quary diff operation that just performs steps 1-4 above and outputs the diff between project and target, without handling the dropping of stale objects. This would still provide visibility to users about stale objects.

@louisjoecodes louisjoecodes added the enhancement New feature or request label May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant