An Ubuntu container for Apache Mesos and Marathon on CoreOS. You can use Deimos in conjunction with Marathon and Mesos to run Docker containers on the CoreOS host that is hosting this container. For more info on Mesos and Marathon, please visit mesosphere.io
This container has a basic install of Zookeeper, Mesos, Marathon and Deimos. It kick starts a mesos_bootstrap.sh
script to configure all the components. For this, it needs some environment variables to be passed in using the -e
flag.
The MAIN_IP
and DOCKER0_IP
are required and have no default.
$MAIN_IP - the IP of the CoreOS host running Docker to which Mesos master and slave can bind (required)
$DOCKER0_IP - the IP assigned to the docker0 interface on the CoreOS host (required)
$ETCD_PORT - the port on which ETCD runs on CoreOS (default: 4001)
$ETCD_MESOS_PATH - the path in ETCD where we store Mesos related data (default: /mesos)
$ETCD_TTL - the TTL used in retrying ETCD calls in seconds (default: 10)
This container relies on a working ETCD connection, typically used with CoreOS.
When no arguments are passed into this script, it will try to dynamically configure a Mesos cluster consisting of:
- 1 node running a Master, Zookeeper, Marathon and a local slave. Marathon runs in a separate docker container.
- x slave nodes, depending on the amount of nodes you spin up. The slaves only run the Mesos slave process and Deimos.
Discovery of the Master's IP is done using ETCD. For this to work, all nodes should be in
the same ETCD cluster.
If automagic setup doesn't work, you can also pass in arguments and the --etcd=false
flag to set up Mesos manually.
For example, when you want to start up the whole shebang using auto-discovery.
docker run --rm --name mesos \
--net=host \
-p 5050:5050 \
-p 5051:5051 \
-p 2181:2181 \
-e MAIN_IP=172.17.8.101 \
-e DOCKER0_IP=`ifconfig docker0 | grep 'inet ' | awk '{print $2}'` \
-v /var/lib/docker/btrfs/subvolumes:/var/lib/docker/btrfs/subvolumes \
-v /var/run/docker.sock:/var/run/docker.sock \
tnolet/mesos-on-coreos:1.0
Notice we are passing in the PUBLIC_IP
environment variable and dynamically grabbing the docker0 IP.
Also, we do not run in bridged mode but use the hosts IP
network stack. This is important for Mesos masters and slaves for reporting the hosts IP address using Zookeeper.
After the whole thing is started up, the normal API and dashboards are available at the master's IP, e.g
172.17.8.101:5050
for the Mesos dashboard and 172.17.8.101:8080
for Marathon.
When using ETCD for auto-discovery, you need to first start a master passing in the --etcd=false
flag. This
will start a master and a zookeeper instance in the same container
...
tnolet/mesos-on-coreos:1.0 master --etcd=false
Then start a Marathon container, passing in the zookeeper address for the master. No need to specify the extra etcd flag.
...
tnolet/mesos-on-coreos:1.0 master marathon --master=zk:https://172.17.8.101:2181/mesos
Then start a slave container, passing in the zookeeper address for the master. You can boot up as many slaves as you want on as many containers. As longs as they are reachable over the network.
...
tnolet/mesos-on-coreos:1.0 master slave --master=zk:https://172.17.8.101:2181/mesos
CoreOS uses systemd
and cloudconfig
to control running services on startup. You can find examples of the above
commands in the a handy user-data.yml
which you can use with CoreOS to get all this running instantly at boot.
You can find it in the https://github.com/tnolet/mesos_on_coreos repo.
In the https://github.com/tnolet/mesos_on_coreos repo you can also find
a pre-configured Vagrantfile and AWS Cloudformation template (aws_cfn_template.json). Passing in a fresh ETCD discovery URL into either using
the user-data.yml
should get you up and running very fast.
Set the num_instances
option to the total amount of boxes you want to spin up. This includes the master. You can also
set the memory and amount of CPU's if you like.
# Vagrantfile
CLOUD_CONFIG_PATH = "./user-data.yml"
$num_instances = 3
$vb_memory = 1024
$vb_cpus = 1
Open the accompanying user-data.yml
an paste in a discovery URL from ETCD (found here: http:https://discovery.etcd.io/new)
#user-data.yml
coreos:
etcd:
#generate a new token for each unique cluster from https://discovery.etcd.io/new
discovery: https://discovery.etcd.io/27f2c10f29cd24a466a634aaabf64b2d
Then do a vagrant up
and grab some coffee. It will downloaded CoreOS once and the mesos-on-coreos docker container for
each box.
If you are fed up with downloading containers when developing, save the container to disk and import it when you are booting your Vagrant boxes. It saves a lot of time when you are destroying your Vagrant boxes a lot.
$ docker save tnolet/mesos-on-coreos > tnolet_mesos-on-coreos.tar
Just mount your disk to Vagrant and import it using this snippet in your Vagrantfile:
DOCKER_UBUNTU_MESOS="tnolet_mesos-on-coreos.tar"
config.vm.synced_folder ".", "/home/core/share", id: "core", :nfs => true, :mount_options => ['nolock,vers=3,udp']
config.vm.provision :shell, :inline => "export TMPDISK=/", :privileged => false
if File.exist?(DOCKER_UBUNTU_MESOS)
config.vm.provision :shell, :inline => "docker load -i /home/core/share/#{DOCKER_UBUNTU_MESOS}", :privileged => false
end
This was tested on OSX Mavericks.
- replace flags with REAL flags that don't depend on the position in cmd line