Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support different types for CompatReader #3745

Merged
merged 7 commits into from
Apr 24, 2024

Conversation

KKould
Copy link
Collaborator

@KKould KKould commented Apr 19, 2024

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

#3517

What's changed and what's your intention?

Support CompatReader to be compatible with different types of reads (for modify column type)

Whether the type allows conversion will be verified in the next pr: (reference:
ModifyColumn::validate https://github.com/GreptimeTeam/greptimedb/pull/3701/files#diff-839e23ed322c90eacc42fdddeb1f7108335e6434eba0a7a1c6dbc42052a8b9e9)

Tips: Please closely 64FD8R166 in the cast in this line of code, the safe of the CastOptions it used by default is true, so it is guaranteed that the type is allowed to be transformed and specific When the value cannot be converted, set the value to null, e.g. type: String -> Integer, value: "1" -> 1, "hello" -> null

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

@github-actions github-actions bot added the docs-not-required This change does not impact docs. label Apr 19, 2024
Copy link

codecov bot commented Apr 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.28%. Comparing base (4437225) to head (089cd05).
Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3745      +/-   ##
==========================================
- Coverage   85.51%   85.28%   -0.24%     
==========================================
  Files         946      946              
  Lines      159555   160034     +479     
==========================================
+ Hits       136447   136482      +35     
- Misses      23108    23552     +444     

@KKould
Copy link
Collaborator Author

KKould commented Apr 21, 2024

We can also set up a simple rule.

  • fill NULL if the column is nullable
  • fill default if the column is not null

#3517 (comment)

cast can only be converted to null but cannot set default. Do you have any suggestions on this?

@evenyag
Copy link
Contributor

evenyag commented Apr 22, 2024

cast can only be converted to null but cannot set default. Do you have any suggestions on this?

Maybe we could only allow modifying the data type when the column is nullable or casting the type won't be fallible.

Copy link
Contributor

@evenyag evenyag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the slow review. Don't have enough time to review this last week.

src/mito2/src/read/compat.rs Outdated Show resolved Hide resolved
src/mito2/src/read/compat.rs Show resolved Hide resolved
src/mito2/src/read/compat.rs Outdated Show resolved Hide resolved
src/mito2/src/read/compat.rs Outdated Show resolved Hide resolved
- add unit test `test_safe_cast_to_null` to test safely cast
- add DataType to projected_fields
- remove TODO
@KKould KKould requested a review from a team as a code owner April 22, 2024 05:03
@KKould
Copy link
Collaborator Author

KKould commented Apr 22, 2024

Maybe we could only allow modifying the data type when the column is nullable or casting the type won't be fallible.

I added this check in pr: #3757 9003afc

casting the type won't be fallible

I think this is difficult to do. Because when conversion is allowed between certain types, the value may not necessarily be able to be converted. Of course, conversion between some types is definitely successful(to String), but judging by self will be relatively intrusive (and prone to bugs)

so I chose to check if column is allowed to be nullable

Copy link
Contributor

@evenyag evenyag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM

src/mito2/src/read/compat.rs Outdated Show resolved Hide resolved
src/mito2/src/read/compat.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@evenyag evenyag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@evenyag evenyag enabled auto-merge April 23, 2024 13:40
Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

src/mito2/src/read/compat.rs Show resolved Hide resolved
@evenyag evenyag added this pull request to the merge queue Apr 24, 2024
Copy link
Contributor

@v0y4g3r v0y4g3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Merged via the queue into GreptimeTeam:main with commit 42e7403 Apr 24, 2024
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required This change does not impact docs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants