Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Enable run_id overrides for Checkpoint and ValidationDefinition #9760

Merged
merged 7 commits into from
Apr 15, 2024

Conversation

cdkini
Copy link
Member

@cdkini cdkini commented Apr 15, 2024

Users should be able to pass a custom run_id for validation workflows.

PR also contains some misc fixes around metadata to enable proper rendering. Finally, there is an update to the type annotation of Checkpoint to suppress a warning we see in the root validator of EmailAction (not the best practice but we'll return to resolve this soon).

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented Apr 15, 2024

Deploy Preview for niobium-lead-7998 ready!

Name Link
🔨 Latest commit 1ea7f67
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/661d81928238d40008e2e47a
😎 Deploy Preview https://deploy-preview-9760.docs.greatexpectations.io
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

) -> ExpectationSuiteValidationResult:
validator = Validator(
batch_definition=self.batch_definition,
batch_parameters=batch_parameters,
result_format=result_format,
)
results = validator.validate_expectation_suite(self.suite, suite_parameters)
results.meta["run_id"] = run_id
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should never be looking into meta - this should be changed

@@ -95,7 +95,7 @@ def render(
):
# Gather run identifiers
run_name, run_time = self._parse_run_values(validation_results)
expectation_suite_name = validation_results.meta["expectation_suite_name"]
expectation_suite_name = validation_results.suite_name
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New top level property allows this to be a much safer retrieval

Copy link

codecov bot commented Apr 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.68%. Comparing base (f293aa2) to head (1ea7f67).
Report is 1 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9760      +/-   ##
===========================================
- Coverage    82.69%   82.68%   -0.01%     
===========================================
  Files          512      512              
  Lines        46901    46902       +1     
===========================================
- Hits         38783    38782       -1     
- Misses        8118     8120       +2     
Flag Coverage Δ
3.10 65.03% <50.00%> (-0.01%) ⬇️
3.11 65.03% <50.00%> (-0.01%) ⬇️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.73% <33.33%> (-0.01%) ⬇️
3.11 aws_deps 48.84% <0.00%> (-0.01%) ⬇️
3.11 big 63.93% <33.33%> (-0.01%) ⬇️
3.11 databricks 48.03% <33.33%> (-0.01%) ⬇️
3.11 filesystem 63.69% <100.00%> (-0.03%) ⬇️
3.11 mssql 47.24% <0.00%> (-0.01%) ⬇️
3.11 mysql 47.29% <0.00%> (-0.01%) ⬇️
3.11 postgresql 54.02% <33.33%> (-0.01%) ⬇️
3.11 snowflake 48.55% <33.33%> (-0.01%) ⬇️
3.11 spark 60.42% <33.33%> (-0.01%) ⬇️
3.11 trino 53.65% <33.33%> (-0.01%) ⬇️
3.8 65.04% <50.00%> (-0.01%) ⬇️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.74% <33.33%> (-0.01%) ⬇️
3.8 aws_deps 48.85% <0.00%> (-0.01%) ⬇️
3.8 big 63.93% <33.33%> (-0.01%) ⬇️
3.8 databricks 48.04% <33.33%> (-0.01%) ⬇️
3.8 filesystem 63.69% <100.00%> (-0.03%) ⬇️
3.8 mssql 47.22% <0.00%> (-0.01%) ⬇️
3.8 mysql 47.28% <0.00%> (-0.01%) ⬇️
3.8 postgresql 54.01% <33.33%> (-0.01%) ⬇️
3.8 snowflake 48.57% <33.33%> (-0.01%) ⬇️
3.8 spark 60.38% <33.33%> (-0.01%) ⬇️
3.8 trino 53.64% <33.33%> (-0.01%) ⬇️
3.9 65.04% <50.00%> (-0.01%) ⬇️
cloud 0.00% <0.00%> (ø)
docs-basic 54.30% <33.33%> (-0.01%) ⬇️
docs-creds-needed 54.86% <33.33%> (-0.01%) ⬇️
docs-spark 54.40% <33.33%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cdkini cdkini enabled auto-merge April 15, 2024 19:35
@cdkini cdkini added this pull request to the merge queue Apr 15, 2024
Merged via the queue into develop with commit 7ecf3cf Apr 15, 2024
73 of 74 checks passed
@cdkini cdkini deleted the b/_/patch_checkpoint_run_id_issues branch April 15, 2024 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants