[FLINK-11064] [table] Setup a new flink-table module structure

This commit splits the flink-table module into multiple submodules in accordance with FLIP-32 (step 1). The new module structure looks as follows: flink-table-common ^ | flink-table-api-java <------- flink-table-api-scala ^ ^ | | flink-table-api-java-bridge flink-table-api-scala-bridge ^ | flink-table-planner The module structure assumes that the type system has been reworked such that only one table environment exists for both Java and Scala users. The module `flink-table-planner` contains the content of the old `flink-table` module. From there we can distribute ported classes to their final module without breaking backwards compatibility or force users to update their dependencies again. For example, if a user wants to implement a pure table program in Scala, `flink-table-api-scala` and `flink-table-planner` need to be added to the project. Until we support pure table programs, `flink-table-api-scala/java-bridge` and `flink-table-planner` need to be added to the project. This closes apache#7587.
Pance · Jan 31, 2019 · 0ab1549 · 0ab1549
1 parent 0e94ecf
commit 0ab1549
Show file tree

Hide file tree

Showing 883 changed files with 658 additions and 88 deletions.
diff --git a/docs/dev/table/index.md b/docs/dev/table/index.md
@@ -31,41 +31,78 @@ The Table API and the SQL interfaces are tightly integrated with each other as w
 
 **Please note that the Table API and SQL are not yet feature complete and are being actively developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
 
-Setup
------
+Dependency Structure
+--------------------
 
-The Table API and SQL are bundled in the `flink-table` Maven artifact. 
-The following dependency must be added to your project in order to use the Table API and SQL:
+All Table API and SQL components are bundled in the `flink-table` Maven artifact.
+
+The following dependencies are relevant for most projects:
+
+* `flink-table-common`: A common module for extending the table ecosystem by custom functions, formats, etc.
+* `flink-table-api-java`: The Table & SQL API for pure table programs using the Java programming language (in early development stage, not recommended!).
+* `flink-table-api-scala`: The Table & SQL API for pure table programs using the Scala programming language (in early development stage, not recommended!).
+* `flink-table-api-java-bridge`: The Table & SQL API with DataStream/DataSet API support using the Java programming language.
+* `flink-table-api-scala-bridge`: The Table & SQL API with DataStream/DataSet API support using the Scala programming language.
+* `flink-table-planner`: The table program planner and runtime.
+* `flink-table-uber`: Packages the modules above into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table*.jar` is located in the `/opt` directory of a Flink release and can be moved to `/lib` if desired.
+
+### Table Program Dependencies
+
+The following dependencies must be added to a project in order to use the Table API & SQL for defining pipelines:
 
 {% highlight xml %}
 <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
+ <artifactId>flink-table-planner{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version}}</version>
 </dependency>
 {% endhighlight %}
 
-In addition, you need to add a dependency for either Flink's Scala batch or streaming API. For a batch query you need to add:
+Additionally, depending on the target programming language, you need to add the Java or Scala API.
 
 {% highlight xml %}
+<!-- Either... -->
 <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-scala{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
+ <artifactId>flink-table-api-java-bridge{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version}}</version>
+</dependency>
+<!-- or... -->
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-api-scala-bridge{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version}}</version>
 </dependency>
 {% endhighlight %}
 
-For a streaming query you need to add:
+Internally, parts of the table ecosystem are implemented in Scala. Therefore, please make sure to add the following dependency for both batch and streaming applications:
 
 {% highlight xml %}
 <dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
+ <version>{{site.version}}</version>
+</dependency>
+{% endhighlight %}
+
+### Extension Dependencies
+
+If you want to implement a [custom format](({{ site.baseurl }}/dev/table/sourceSinks.html#define-a-tablefactory)) for interacting with Kafka or a set of [user-defined functions]({{ site.baseurl }}/dev/table/functions.html), the following dependency is sufficient and can be used for JAR files for the SQL Client:
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-common</artifactId>
+ <version>{{site.version}}</version>
 </dependency>
 {% endhighlight %}
 
-**Note:** Due to an issue in Apache Calcite, which prevents the user classloaders from being garbage-collected, we do *not* recommend building a fat-jar that includes the `flink-table` dependency. Instead, we recommend configuring Flink to include the `flink-table` dependency in the system classloader. This can be done by copying the `flink-table.jar` file from the `./opt` folder to the `./lib` folder. See [these instructions]({{ site.baseurl }}/dev/linking.html) for further details.
+Currently, the module includes extension points for:
+- `SerializationSchemaFactory`
+- `DeserializationSchemaFactory`
+- `ScalarFunction`
+- `TableFunction`
+- `AggregateFunction`
 
 {% top %}
 
@@ -80,4 +117,4 @@ Where to go next?
 * [Built-in Functions]({{ site.baseurl }}/dev/table/functions.html): Supported functions in Table API and SQL.
 * [SQL Client]({{ site.baseurl }}/dev/table/sqlClient.html): Play around with Flink SQL and submit a table program to a cluster without programming knowledge.
 
-{% top %}
+{% top %}
diff --git a/flink-connectors/flink-connector-cassandra/pom.xml b/flink-connectors/flink-connector-cassandra/pom.xml
@@ -154,6 +154,26 @@ under the License.
  <version>${guava.version}</version>
  </dependency>
 
+ <!-- Table ecosystem -->
+ <!-- Projects depending on this project won't depend on flink-table-*. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+ <!-- A planner dependency won't be necessary once FLIP-32 has been completed. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+
+ <!-- Test dependencies -->
+
  <dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-runtime_${scala.binary.version}</artifactId>
@@ -196,11 +216,5 @@ under the License.
  </exclusion>
  </exclusions>
  </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
- <version>${project.version}</version>
- <scope>provided</scope>
- </dependency>
  </dependencies>
 </project>
diff --git a/flink-connectors/flink-connector-elasticsearch-base/pom.xml b/flink-connectors/flink-connector-elasticsearch-base/pom.xml
@@ -70,13 +70,21 @@ under the License.
  </exclusions>
  </dependency>
 
- <!-- Used for the Elasticsearch table sink. -->
+ <!-- Table ecosystem -->
+ <!-- Projects depending on this project won't depend on flink-table-*. -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+ <!-- A planner dependency won't be necessary once FLIP-32 has been completed. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <scope>provided</scope>
- <!-- Projects depending on this project, won't depend on flink-table. -->
  <optional>true</optional>
  </dependency>
 
@@ -108,7 +116,7 @@ under the License.
  <!-- Elasticsearch table descriptor testing -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <type>test-jar</type>
  <scope>test</scope>

diff --git a/flink-connectors/flink-connector-elasticsearch6/pom.xml b/flink-connectors/flink-connector-elasticsearch6/pom.xml
@@ -84,13 +84,21 @@ under the License.
  <version>2.9.1</version>
  </dependency>
 
- <!-- Used for the Elasticsearch table sink. -->
+ <!-- Table ecosystem -->
+ <!-- Projects depending on this project won't depend on flink-table-*. -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+ <!-- A planner dependency won't be necessary once FLIP-32 has been completed. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <scope>provided</scope>
- <!-- Projects depending on this project, won't depend on flink-table. -->
  <optional>true</optional>
  </dependency>
 
@@ -158,7 +166,7 @@ under the License.
  <!-- Elasticsearch table descriptor testing -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <type>test-jar</type>
  <scope>test</scope>

diff --git a/flink-connectors/flink-connector-kafka-0.10/pom.xml b/flink-connectors/flink-connector-kafka-0.10/pom.xml
@@ -77,12 +77,21 @@ under the License.
  <version>${kafka.version}</version>
  </dependency>
 
+ <!-- Table ecosystem -->
+ <!-- Projects depending on this project won't depend on flink-table-*. -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+ <!-- A planner dependency won't be necessary once FLIP-32 has been completed. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <scope>provided</scope>
- <!-- Projects depending on this project, won't depend on flink-table. -->
  <optional>true</optional>
  </dependency>
 
@@ -166,7 +175,7 @@ under the License.
 
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <type>test-jar</type>
  <scope>test</scope>

diff --git a/flink-connectors/flink-connector-kafka-0.11/pom.xml b/flink-connectors/flink-connector-kafka-0.11/pom.xml
@@ -77,12 +77,21 @@ under the License.
  <version>${kafka.version}</version>
  </dependency>
 
+ <!-- Table ecosystem -->
+ <!-- Projects depending on this project won't depend on flink-table-*. -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+ <!-- A planner dependency won't be necessary once FLIP-32 has been completed. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <scope>provided</scope>
- <!-- Projects depending on this project, won't depend on flink-table. -->
  <optional>true</optional>
  </dependency>
 
@@ -174,7 +183,7 @@ under the License.
 
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <type>test-jar</type>
  <scope>test</scope>

diff --git a/flink-connectors/flink-connector-kafka-0.8/pom.xml b/flink-connectors/flink-connector-kafka-0.8/pom.xml
@@ -72,12 +72,21 @@ under the License.
  </exclusions>
  </dependency>
 
+ <!-- Table ecosystem -->
+ <!-- Projects depending on this project won't depend on flink-table-*. -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+ <!-- A planner dependency won't be necessary once FLIP-32 has been completed. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <scope>provided</scope>
- <!-- Projects depending on this project, won't depend on flink-table. -->
  <optional>true</optional>
  </dependency>
 
@@ -182,7 +191,7 @@ under the License.
 
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <type>test-jar</type>
  <scope>test</scope>

diff --git a/flink-connectors/flink-connector-kafka-0.9/pom.xml b/flink-connectors/flink-connector-kafka-0.9/pom.xml
@@ -67,12 +67,21 @@ under the License.
  </exclusions>
  </dependency>
 
+ <!-- Table ecosystem -->
+ <!-- Projects depending on this project won't depend on flink-table-*. -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+ <!-- A planner dependency won't be necessary once FLIP-32 has been completed. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <scope>provided</scope>
- <!-- Projects depending on this project, won't depend on flink-table. -->
  <optional>true</optional>
  </dependency>
 
@@ -139,7 +148,7 @@ under the License.
 
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <type>test-jar</type>
  <scope>test</scope>

diff --git a/flink-connectors/flink-connector-kafka-base/pom.xml b/flink-connectors/flink-connector-kafka-base/pom.xml
@@ -57,12 +57,21 @@ under the License.
  <scope>provided</scope>
  </dependency>
 
+ <!-- Table ecosystem -->
+ <!-- Projects depending on this project won't depend on flink-table-*. -->
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <optional>true</optional>
+ </dependency>
+ <!-- A planner dependency won't be necessary once FLIP-32 has been completed. -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <scope>provided</scope>
- <!-- Projects depending on this project, won't depend on flink-table. -->
  <optional>true</optional>
  </dependency>
 
@@ -183,7 +192,7 @@ under the License.
 
  <dependency>
  <groupId>org.apache.flink</groupId>
- <artifactId>flink-table_${scala.binary.version}</artifactId>
+ <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${project.version}</version>
  <type>test-jar</type>
  <scope>test</scope>