Skip to content

Commit

Permalink
Merge pull request apache#11853 Update multi-language roadmap status.
Browse files Browse the repository at this point in the history
  • Loading branch information
robertwb authored May 28, 2020
2 parents 74962ce + e7abbdb commit 1e7d194
Showing 1 changed file with 10 additions and 9 deletions.
19 changes: 10 additions & 9 deletions website/www/site/content/en/roadmap/connectors-multi-sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ efforts. See [Beam portability framework roadmap](https://beam.apache.org/roadma

# Cross-language transforms

_Last updated on November 2019._
_Last updated on May 2020._

As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits.

* Connector sharing across SDKs. For example,
+ Beam pipelines written using Python and Go SDKs will be able to utilize the vast selection of connectors that are currently implemented for Java SDK.
+ Java SDK will be able to utilize connectors for systems that only offer a Python API.
+ Go SDK, will be able to utilize connectors currently available for Java and Python SDKs.
* Ease of developing and maintaining Beam transforms - in general, with cross-language transforms, Beam transform authors will be able to implement new Beam transforms using a
* Ease of developing and maintaining Beam transforms - in general, with cross-language transforms, Beam transform authors will be able to implement new Beam transforms using a
language of choice and utilize these transforms from other languages reducing the maintenance and support overheads.
* [Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/), that is currently only available to Java SDK, will become available to Python and Go SDKs.
* [Beam TFX transforms](https://www.tensorflow.org/tfx/transform/get_started), that are currently only available to Beam Python SDK pipelines will become available to Java and Go SDKs.
Expand All @@ -48,9 +48,9 @@ Work related to developing/updating the cross-language transforms API for Java/P

* Basic API for Java SDK - completed
* Basic API for Python SDK - completed
* Basic API for Go SDK - Not started
* Basic API for Go SDK - In progress
* Basic cross-language transform expansion service for Java and Python SDKs - completed
* Artifact staging - In progress - [email thread](https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E), [doc](https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8)
* Artifact staging - mostly completed - [email thread](https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E), [doc](https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8)

### Support for Flink runner

Expand All @@ -62,27 +62,29 @@ Work related to making cross-language transforms available for Flink runner.

Work related to making cross-language transforms available for Dataflow runner.

* Basic support for executing cross-language transforms on Dataflow runner
* Basic support for executing cross-language transforms on Dataflow runner
+ This work requires updates to Dataflow service's job submission and job execution logic. This is currently being developed at Google.

### Support for Direct runner

Work related to making cross-language transforms available on Direct runner

* Basic support for executing cross-language transforms on portable Direct runner - Not started
* Basic support for executing cross-language transforms on Pyton Direct runner - completed
* Basic support for executing cross-language transforms on Java Direct runner - Not started

### Connector/transform support

Ongoing and planned work related to making existing connectors/transforms available to other SDKs through the cross-language transforms framework.

* Java KafkIO - In progress - [BEAM-7029](https://issues.apache.org/jira/browse/BEAM-7029)
* Java KafkaIO - completed - [BEAM-7029](https://issues.apache.org/jira/browse/BEAM-7029)
* Java PubSubIO - In progress - [BEAM-7738](https://issues.apache.org/jira/browse/BEAM-7738)
* Java SQL - completed - [BEAM-8603](https://issues.apache.org/jira/browse/BEAM-8603)

### Portable Beam schema

Portable Beam schema support will provide a generalized mechanism for serializing and transferring data across language boundaries which will be extremely useful for pipelines that employ cross-language transforms.

* Make row coder a standard coder and implement in python - In progress - [BEAM-7886](https://issues.apache.org/jira/browse/BEAM-7886)
* Make row coder a standard coder and implement in python - completed - [BEAM-7886](https://issues.apache.org/jira/browse/BEAM-7886)

### Integration/Performance testing

Expand All @@ -94,4 +96,3 @@ Work related to adding documenting on cross-language transforms to Beam Website.

* Document cross-language transforms API for Java/Python - Not started
* Document API for making existing transforms available as cross-language transforms for Java/Python - Not started

0 comments on commit 1e7d194

Please sign in to comment.