Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pipeline): transform support on_failure #4123

Merged
merged 61 commits into from
Jun 17, 2024
Merged

Conversation

yuanbohan
Copy link
Contributor

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

when transform failed, the on_failure will try to prevent the failure ot ETL. And the valid value for on_failure is: default or ignore.

  • default: set to default value
  • ignore: set to None (the same result of no such field to be set)

The following will set the number field to default (which is 0 in the yaml) if its value can not be coerced to int32

trnsform:
  field: number
  type: int32
  default: 0
  on_failure: default

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.

paomian and others added 30 commits May 27, 2024 16:51
refactor: use `inserter` instead of `grpc_handler` in pipeline operator
@yuanbohan yuanbohan requested a review from a team as a code owner June 8, 2024 13:37
@github-actions github-actions bot added the docs-not-required This change does not impact docs. label Jun 8, 2024
@yuanbohan yuanbohan changed the title feat(pipeline): transform on_failure feat(pipeline): transform support on_failure Jun 8, 2024
@yuanbohan yuanbohan changed the title feat(pipeline): transform support on_failure feat(pipeline): transform support on_failure Jun 8, 2024
@yuanbohan yuanbohan changed the title feat(pipeline): transform support on_failure feat(pipeline): transform support on_failure Jun 8, 2024
Copy link

codecov bot commented Jun 8, 2024

Codecov Report

Attention: Patch coverage is 51.42857% with 85 lines in your changes missing coverage. Please review.

Project coverage is 84.89%. Comparing base (2faa6d6) to head (83e59b1).
Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4123      +/-   ##
==========================================
- Coverage   84.94%   84.89%   -0.06%     
==========================================
  Files        1018     1022       +4     
  Lines      178688   179152     +464     
==========================================
+ Hits       151780   152083     +303     
- Misses      26908    27069     +161     

@shuiyisong
Copy link
Contributor

Wait for #4014 get merged first, then we can add some test.

Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Good job!

src/pipeline/src/etl/transform/mod.rs Show resolved Hide resolved
src/pipeline/src/etl/transform/mod.rs Show resolved Hide resolved
@shuiyisong shuiyisong added this pull request to the merge queue Jun 17, 2024
Merged via the queue into main with commit 0aceebf Jun 17, 2024
50 checks passed
@shuiyisong shuiyisong deleted the transform-on-failure branch June 17, 2024 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required This change does not impact docs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants