[FLINK-14526][hive] Support Hive version 1.1.0 and 1.1.1 #9995

lirui-apache · 2019-10-25T11:41:03Z

What is the purpose of the change

To support Hive 1.1.0 and 1.1.1

Brief change log

Added two profiles and two shim implementations for 1.1.0 and 1.1.1 respectively.
1.1.x HMS always tries to compute some table stats during altering a non-partitioned table, which can override what we put in the table parameters. Therefore altering non-partitioned table stats for 1.1.x is disabled.
DATE column stats are not supported in 1.1.x. Therefore shim methods are added to handle these column stats.
Made HiveShim serializable because it's needed in HiveGenericUDF and HiveGenericUDTF. This change should have been made before this PR. We were good only because there's no test to expose this issue.
Java constant object inspectors are not available in 1.1.x. Therefore added a shim method to get constant object inspectors. And because of this, we have to handle Writables for Hive UDFs if they're initialized with some constant arguments. Added a conversion from Hive Java primitive to Hive Writable primitive for this.
1.1.x HCatalog artifacts are built with Hadoop-1 and incompatible with Hadoop-2. As a result, we cannot use the Hive runner to generate table data during tests. Added a util method to serve this purpose.
For features unsupported by 1.1.x, related test cases are disabled for these old Hive versions.

Verifying this change

Covered by existing test cases.
Manually verified by running mvn verify -Phive-1.1.0 and mvn verify -Phive-1.1.1.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): yes
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? NA

lirui-apache · 2019-10-25T11:42:02Z

cc @xuefuz @bowenli86 @zjuwangg

flinkbot · 2019-10-25T11:46:52Z

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 2f28e38 (Wed Dec 04 14:51:54 UTC 2019)

Warnings:

1 pom.xml files were touched: Check for build and licensing issues.
No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

❓ 1. The [description] looks good.
❓ 2. There is [consensus] that the contribution should go into to Flink.
❓ 3. Needs [attention] from.
❓ 4. The change fits into the overall [architecture].
❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
@flinkbot approve all to approve all aspects
@flinkbot approve-until architecture to approve everything until architecture
@flinkbot attention @username1 [@username2 ..] to require somebody's attention
@flinkbot disapprove architecture to remove an approval you gave earlier

flinkbot · 2019-10-25T11:51:23Z

CI report:

5f0728f : SUCCESS Build
fe532e1 : SUCCESS Build
959bd5e : SUCCESS Build
2f28e38 : SUCCESS Build

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run travis re-run the last Travis build

bowenli86

LGTM.

Btw, can you confirm all the test profiles have passed with this change?

bowenli86 · 2019-10-25T17:37:27Z

...nk-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimV120.java

+ String className;
+ switch (primitiveTypeInfo.getPrimitiveCategory()) {
+ case BOOLEAN:
+ className = "org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaConstantBooleanObjectInspector";


curious why does 1.1.0 check WritableConstant and 1.1.1 check JavaConstant?

Because java constant object inspectors are not available until 1.2.0

xuefuz

The current inheritance pattern is that later version hive shim inherits from the previous version unless there is a compelling reason. This is to reduce code duplication.

However, HiveShimV120 isn't inherited from HiveShimV111.

It's good if @bowenli86 can also review.

flink-connectors/flink-connector-hive/pom.xml

xuefuz · 2019-10-25T17:48:13Z

.../flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShim.java

 import java.util.List;
 import java.util.Map;

 /**
 * A shim layer to support different versions of Hive.
 */
-public interface HiveShim {
+public interface HiveShim extends Serializable {


It's probably better to leave this change (about serialization) to a dedicated PR with relevant tests. It makes review easier and limits the scope of this change.

Test will fail if HiveShim is not serializable. Do you mean we should first open and merge a dedicated PR for this and come back later?

Does the test fail before this PR, as HiveShim is not serializable before. If we need the fix to pass the test, we can put in the same PR to reduce the overhead.

Tests pass before this PR only because we don't have enough test coverage to expose the issue. Before this PR, HiveShim is only used to get object inspectors for DATE and TIMESTAMP columns, while with the PR, HiveShim is needed for all columns. So we need the fix to pass the tests.

...rs/flink-connector-hive/src/test/java/org/apache/flink/table/catalog/hive/HiveTestUtils.java

...ctor-hive/src/main/java/org/apache/flink/table/functions/hive/conversion/HiveInspectors.java

lirui-apache · 2019-10-28T07:24:09Z

PR updated to address comments. Also verified tests pass for all our profiles.
@xuefuz @bowenli86 please take a look. Thanks.

xuefuz

+1 looks good to me on my side.

cleanup

lirui-apache · 2019-10-29T07:50:48Z

Rebased. And noted INTERVAL PrimitiveCategory is not available in 1.1.x. So added our own enum for primitive categories.
@xuefuz @bowenli86 Please let me know if you have further comments. Thanks.

bowenli86 · 2019-10-29T17:27:45Z

LGTM, thanks for your contribution! Merging

rmetzger added the review=description? label Oct 25, 2019

rmetzger added the component=Connectors/Hive label Oct 25, 2019

bowenli86 reviewed Oct 25, 2019

View reviewed changes

xuefuz reviewed Oct 25, 2019

View reviewed changes

lirui-apache force-pushed the FLINK-14526 branch from 5f0728f to fe532e1 Compare October 28, 2019 06:13

xuefuz reviewed Oct 28, 2019

View reviewed changes

lirui-apache added 4 commits October 29, 2019 13:39

[FLINK-14526][hive] Support Hive version 1.1.0 and 1.1.1

1123041

address comments

516ea44

don't check file format when loading data

7921df7

rebase

2f28e38

cleanup

lirui-apache force-pushed the FLINK-14526 branch from 959bd5e to 2f28e38 Compare October 29, 2019 07:45

asfgit closed this in 6cc09ac Oct 29, 2019

lirui-apache deleted the FLINK-14526 branch October 30, 2019 02:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-14526][hive] Support Hive version 1.1.0 and 1.1.1 #9995

[FLINK-14526][hive] Support Hive version 1.1.0 and 1.1.1 #9995

lirui-apache commented Oct 25, 2019

lirui-apache commented Oct 25, 2019

flinkbot commented Oct 25, 2019 •

edited

Loading

flinkbot commented Oct 25, 2019 •

edited

Loading

bowenli86 left a comment

bowenli86 Oct 25, 2019

lirui-apache Oct 28, 2019

xuefuz left a comment

xuefuz Oct 25, 2019

lirui-apache Oct 28, 2019

xuefuz Oct 28, 2019

lirui-apache Oct 29, 2019

lirui-apache commented Oct 28, 2019

xuefuz left a comment •

edited

Loading

lirui-apache commented Oct 29, 2019

bowenli86 commented Oct 29, 2019 •

edited

Loading

[FLINK-14526][hive] Support Hive version 1.1.0 and 1.1.1 #9995

[FLINK-14526][hive] Support Hive version 1.1.0 and 1.1.1 #9995

Conversation

lirui-apache commented Oct 25, 2019

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

lirui-apache commented Oct 25, 2019

flinkbot commented Oct 25, 2019 • edited Loading

Automated Checks

Review Progress

flinkbot commented Oct 25, 2019 • edited Loading

CI report:

bowenli86 left a comment

Choose a reason for hiding this comment

bowenli86 Oct 25, 2019

Choose a reason for hiding this comment

lirui-apache Oct 28, 2019

Choose a reason for hiding this comment

xuefuz left a comment

Choose a reason for hiding this comment

xuefuz Oct 25, 2019

Choose a reason for hiding this comment

lirui-apache Oct 28, 2019

Choose a reason for hiding this comment

xuefuz Oct 28, 2019

Choose a reason for hiding this comment

lirui-apache Oct 29, 2019

Choose a reason for hiding this comment

lirui-apache commented Oct 28, 2019

xuefuz left a comment • edited Loading

Choose a reason for hiding this comment

lirui-apache commented Oct 29, 2019

bowenli86 commented Oct 29, 2019 • edited Loading

flinkbot commented Oct 25, 2019 •

edited

Loading

flinkbot commented Oct 25, 2019 •

edited

Loading

xuefuz left a comment •

edited

Loading

bowenli86 commented Oct 29, 2019 •

edited

Loading