Skip to content

Commit

Permalink
Updated READMEs.
Browse files Browse the repository at this point in the history
  • Loading branch information
hmusavi committed Dec 13, 2018
1 parent f2846e2 commit 2ea5106
Show file tree
Hide file tree
Showing 3 changed files with 123 additions and 2 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,11 +132,11 @@ Read the following guides to familiarize yourself with ODE's Docker and Kafka mo

**Docker**

[README.md](docker/README.md)
[README](docker.md)

**Kafka**

[README.md](docker/kafka/README.md)
[README](kafka.md)

---
### Obtain the Source Code
Expand Down
51 changes: 51 additions & 0 deletions docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@

# Getting Started with Docker
The following sections summarize installation and setup of Docker on your machine.
## Windows
- Enable virtualization in your BIOS
- Install [Docker for Windows](https://store.docker.com/editions/community/docker-ce-desktop-windows)
## Mac OSX
- Enable virtualization in your BIOS
- Install [Docker for Mac](## Windows
- Enable visualization in your BIOS
- Install [Docker for Windows](https://store.docker.com/editions/community/docker-ce-desktop-windows)
)
## Linux
- Install [Docker Engine](https://store.docker.com/search?type=edition&offering=community&operating_system=linux) on Docker host machine
- Install [docker-compose](https://docs.docker.com/compose/install/) on docker host and/or local machine
- Create a ```.env``` file in the same directory as ```docker-compose.yml```.

## Step 1 - Build and deploy the application
Run ```docker-compose up --build -d```

This starts the configured containers.

## Helpful hints

### Obtaining DOCKER_HOST_IP
1. **Windows**: Run `ifconfig`
2. **OSX**: Look for "inet addr:" under the "en0" network interface
3. **Linux**: Look for "inet addr:" under the "eth0" network interface

### Resolve Git SSL issues by disabling SSL verification
- `export GIT_SSL_NO_VERIFY=1`
### Docker commands
- Bring UP (deploy and start) all containers
- `docker-compose up -d`
- Bring DOWN all containers
- `docker-compose down`
- Lists the running containers under docker-compose
- `docker-compose ps`
- Start a container
- `docker-compose start <container>`
- Restart a container
- `docker-compose restart <container>`
- Stop a container
- `docker-compose stop <container>`
- Stop all containers
- `docker-compose stop`

## Troubleshooting

- Docker builds fail with a `wget: bad address` error
- [Docker fix for SSL issues due to corporate network](https://github.com/usdot-jpo-ode/jpo-ode/wiki/Docker-fix-for-SSL-issues-due-to-corporate-network)
70 changes: 70 additions & 0 deletions kafka.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@

[![Docker Pulls](https://img.shields.io/docker/pulls/wurstmeister/kafka.svg)](https://hub.docker.com/r/wurstmeister/kafka/)
[![Docker Stars](https://img.shields.io/docker/stars/wurstmeister/kafka.svg)](https://hub.docker.com/r/wurstmeister/kafka/)
[![](https://badge.imagelayers.io/wurstmeister/kafka:latest.svg)](https://imagelayers.io/?images=wurstmeister/kafka:latest)

# kafka-docker

Dockerfile for [Apache Kafka](http:https://kafka.apache.org/)

The image is available directly from https://registry.hub.docker.com/

## Tutorial

[http:https://wurstmeister.github.io/kafka-docker/](http:https://wurstmeister.github.io/kafka-docker/)

## Helpful Hints
### Pre-Requisites

- install [docker](docker.md)
- modify the ```KAFKA_ADVERTISED_HOST_NAME``` in ```docker-compose.yml``` to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.)
- if you want to customize any Kafka parameters, simply add them as environment variables in ```docker-compose.yml```, e.g. in order to increase the ```message.max.bytes``` parameter set the environment to ```KAFKA_MESSAGE_MAX_BYTES: 2000000```. To turn off automatic topic creation set ```KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'```

### Usage

Start a cluster:

- ```docker-compose up -d ```

### Add more brokers:

The default [docker-compose.yml](https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose.yml) can launch multiple kafka brokers giving each broker a unique port number. Depending on your use case this might not be desirable. If you need to use specific ports and broker ids, modify the `docker-compose` configuration accordingly. To add more brokers, run:
- ```docker-compose scale kafka=3```

The alternative [docker-compose-single-broker.yml](https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose-single-broker.yml) can be used to deploy a single Kafka broker:
- ```docker-compose -f docker-compose-single-broker.yml up```

### Broker IDs

If you don't specify a broker id in your docker-compose file, it will automatically be generated (see [https://issues.apache.org/jira/browse/KAFKA-1070](https://issues.apache.org/jira/browse/KAFKA-1070). This allows scaling up and down. In this case it is recommended to use the ```--no-recreate``` option of docker-compose to ensure that containers are not re-created and thus keep their names and ids.


### Automatically create topics

If you want to have kafka-docker automatically create topics in Kafka during
creation, a ```KAFKA_CREATE_TOPICS``` environment variable can be
added in ```docker-compose.yml```.

Here is an example snippet from ```docker-compose.yml```:

environment:
KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1"

```Topic 1``` will have 1 partition and 3 replicas, ```Topic 2``` will have 1 partition and 1 replica.

### Advertised hostname

You can configure the advertised hostname in different ways

1. explicitly, using ```KAFKA_ADVERTISED_HOST_NAME```
2. via a command, using ```HOSTNAME_COMMAND```, e.g. ```HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"```

When using commands, make sure you review the "Variable Substitution" section in [https://docs.docker.com/compose/compose-file/](https://docs.docker.com/compose/compose-file/)

If ```KAFKA_ADVERTISED_HOST_NAME``` is specified, it takes presendence over ```HOSTNAME_COMMAND```

For AWS deployment, you can use the Metadata service to get the container host's IP:
```
HOSTNAME_COMMAND=wget -t3 -T2 -qO- http:https://169.254.169.254/latest/meta-data/local-ipv4
```
Reference: http:https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

0 comments on commit 2ea5106

Please sign in to comment.