-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-7274] Add DynamicMessage Schema support #10502
[BEAM-7274] Add DynamicMessage Schema support #10502
Conversation
e4a6ec6
to
3a6ca71
Compare
Run Java PreCommit |
R: @reuvenlax this is an implementation for DynamicMessages. It takes into account the new Logical Types and also use RowWithStorage instead of going with the getters. |
04a8580
to
058a636
Compare
058a636
to
97414c6
Compare
@reuvenlax this is ready for review. It has feature parity with the static compiled proto, including all logical types. |
97414c6
to
e352ff4
Compare
Run Java PreCommit |
51f9c9d
to
a83d803
Compare
This branch has been updated to take the new Logical Types into account. |
602869a
to
7e57a39
Compare
7e57a39
to
9a13256
Compare
Run Java PreCommit |
@reuvenlax @iemejia @TheNeuralBit I've tested this PR extensively on Dataflow. This is way more stable than the original DynamicMessage schema implementation. Can I get a LGTM, that I can focus on the options. Thanks. (master is failing, it was green till rebase) |
9a13256
to
0edacbb
Compare
Run Java PreCommit |
Run Java PreCommit |
0edacbb
to
012c45a
Compare
A few comments. |
...a/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
Outdated
Show resolved
Hide resolved
...a/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
Outdated
Show resolved
Hide resolved
...rotobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One small comment, but otherwise LGTM for merge.
TypeDescriptor.of(context.getBaseClass()), | ||
toRowFunction, | ||
fromRowFunction); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see this function being used anywhere. Remove?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I used it in a production pipeline, but as it's not available in the other providers I removed it, (squashed and rebased).
Add DynamicMessage schema support. This is different from generated classes as it uses the proto descriptors. It uses the ProtoDomain as an index for searching embedded messages.
036b831
to
7a0fe93
Compare
Add DynamicMessage schema support. This is different from
generated classes as it uses the proto descriptors. It uses
the ProtoDomain as an index for searching embedded messages.
R: @reuvenlax
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.