forked from apache/flink
-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[FLINK-10001][docs] Add documentation for job cluster deployment on D…
…ocker and K8s [FLINK-10001][docs] Add documentation for job cluster deployment on K8s This closes apache#6561.
- Loading branch information
1 parent
38e12a4
commit 8d88ff5
Showing
6 changed files
with
149 additions
and
58 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,40 +1,74 @@ | ||
# Apache Flink job cluster deployment on docker using docker-compose | ||
# Apache Flink job cluster Docker image | ||
|
||
## Installation | ||
In order to deploy a job cluster on Docker, one needs to create an image which contains the Flink binaries as well as the user code jars of the job to execute. | ||
This directory contains a `build.sh` which facilitates the process. | ||
The script takes a Flink distribution either from an official release, an archive or a local distribution and combines it with the specified job jar. | ||
|
||
Install the most recent stable version of docker | ||
https://docs.docker.com/installation/ | ||
## Installing Docker | ||
|
||
## Build | ||
Install the most recent stable version of [Docker](https://docs.docker.com/installation/). | ||
|
||
Images are based on the official Java Alpine (OpenJDK 8) image. If you want to | ||
build the flink image run: | ||
## Building the Docker image | ||
|
||
build.sh --from-local-dist --job-jar /path/to/job/jar/job.jar --image-name flink:job | ||
Images are based on the official Java Alpine (OpenJDK 8) image. | ||
|
||
If you want to build the container for a specific version of flink/hadoop/scala | ||
you can configure it in the respective args: | ||
Before building the image, one needs to build the user code jars for the job. | ||
Assume that the job jar is stored under `<PATH_TO_JOB_JAR>` | ||
|
||
docker build --build-arg FLINK_VERSION=1.6.0 --build-arg HADOOP_VERSION=28 --build-arg SCALA_VERSION=2.11 -t "flink:1.6.0-hadoop2.8-scala_2.11" flink | ||
If you want to build the Flink image from the version you have checked out locally run: | ||
|
||
## Deploy | ||
build.sh --from-local-dist --job-jar <PATH_TO_JOB_JAR> --image-name <IMAGE_NAME> | ||
|
||
Note that you first need to call `mvn package -pl flink-dist -am` to build the Flink binaries. | ||
|
||
- Deploy cluster and see config/setup log output (best run in a screen session) | ||
If you want to build the Flink image from an archive stored under `<PATH_TO_ARCHIVE>` run: | ||
|
||
docker-compose up | ||
build.sh --from-archive <PATH_TO_ARCHIVE> --job-jar <PATH_TO_JOB_JAR> --image-name <IMAGE_NAME> | ||
|
||
- Deploy as a daemon (and return) | ||
If you want to build the Flink image for a specific version of Flink/Hadoop/Scala run: | ||
|
||
docker-compose up -d | ||
build.sh --from-release --flink-version 1.6.0 --hadoop-version 2.8 --scala-version 2.11 --image-name <IMAGE_NAME> | ||
|
||
The script will try to download the released version from the Apache archive. | ||
|
||
- Scale the cluster up or down to *N* TaskManagers | ||
## Deploying via Docker compose | ||
|
||
The `docker-compose.yml` contains the following parameters: | ||
|
||
* `FLINK_DOCKER_IMAGE_NAME` - Image name to use for the deployment (default: `flink-job:latest`) | ||
* `FLINK_JOB` - Name of the Flink job to execute (default: none) | ||
* `DEFAULT_PARALLELISM` - Default parallelism with which to start the job (default: 1) | ||
* `FLINK_JOB_ARGUMENTS` - Additional arguments which will be passed to the job cluster (default: none) | ||
|
||
The parameters can be set by exporting the corresponding environment variable. | ||
|
||
Deploy cluster and see config/setup log output (best run in a screen session) | ||
|
||
FLINK_DOCKER_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME> docker-compose up | ||
|
||
Deploy as a daemon (and return) | ||
|
||
FLINK_DOCKER_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME> docker-compose up -d | ||
In order to start the job with a different default parallelism set `DEFAULT_PARALLELISM`. | ||
This will automatically start `DEFAULT_PARALLELISM` TaskManagers: | ||
|
||
FLINK_DOCKER_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME> DEFAULT_PARALLELISM=<DEFAULT_PARALLELISM> docker-compose up | ||
|
||
One can also provide additional job arguments via `FLINK_JOB_ARGUMENTS` which are passed to the job: | ||
|
||
FLINK_DOCKER_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME> FLINK_JOB_ARGUMENTS=<JOB_ARGUMENTS> docker-compose up | ||
|
||
Scale the cluster up or down to *N* TaskManagers | ||
|
||
docker-compose scale taskmanager=<N> | ||
|
||
- Access the Job Manager container | ||
Access the Job Manager container | ||
|
||
docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh | ||
Access the web UI by going to `<IP_DOCKER_MACHINE>:8081` in your web browser. | ||
|
||
- Kill the cluster | ||
Kill the cluster | ||
|
||
docker-compose kill |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters