Caliban is a tool that helps researchers launch and track their numerical experiments in an isolated, reproducible computing environment. It was developed by machine learning researchers and engineers, and makes it easy to go from a simple prototype running on a workstation to thousands of experimental jobs running on Cloud.
With Caliban, you can:
- Develop your experimental code locally and test it inside an isolated (Docker) environment
- Easily sweep over experimental parameters
- Submit your experiments as Cloud jobs, where they will run in the same isolated environment
- Control and keep track of jobs
Install Docker, make sure it's running, then install Caliban (you'll need Python >= 3.6):
pip install caliban
Train a simple deep learning model on your local machine:
git clone https://github.com/google/caliban.git && cd caliban/tutorials/basic
caliban run --nogpu mnist.py
Sweep over learning rates to find the best one (flags are specified in JSON format):
echo '{"learning_rate": [0.01, 0.001, 0.0001]}' | caliban run --experiment_config stdin --nogpu mnist.py
Next:
- See how to submit the experiment to Cloud and use other Caliban features in "Getting Started with Caliban"
- See Installation for detailed installation instructions
- Read the Command Overview for info on Caliban commands.
Full documentation for Caliban lives at Read The Docs.
“Be not afeard; the isle is full of noises,
Sounds, and sweet airs, that give delight and hurt not.
Sometimes a thousand twangling instruments
Will hum about mine ears; and sometime voices,
That, if I then had waked after long sleep,
Will make me sleep again: and then, in dreaming,
The clouds methought would open, and show riches
Ready to drop upon me; that, when I waked,
I cried to dream again.”-- Shakespeare, The Tempest
Caliban's prequisites are Docker and Python >= 3.6.
Make sure your Python is up to date:
$ python --version
Python 3.6.9 # should be >=3.6.0
If not, visit "Installing Python 3.6" before proceeding.
Next, install Caliban via pip:
pip install -U caliban
check if your installation worked by navigating to an empty folder and running
caliban --help
. You should see the usage dialogue:
$ caliban --help
usage: caliban [-h] [--helpfull] [--version]
{shell,notebook,build,run,cloud,cluster,status,stop,resubmit}
...
Caliban executes your code inside a "container", managed by Docker. To get Docker:
- On MacOS, follow the installation instructions at Docker Desktop and start the newly-installed Docker Desktop application.
- On Linux, visit the Docker installation instructions. (It's important that you configure sudo-less Docker and start Docker running on your machine.)
Make sure your Python version is up to date:
$ python --version
Python 3.6.9 # should be >=3.6.0
If you need to upgrade:
- On MacOS, install the latest Python version from python.org (direct link).
- On Linux, run
sudo apt-get update && sudo apt-get install python3.7
.
Caliban's Read the Docs documentation has instructions on:
- Installing the
nvidia-docker2
runtime, so you can use Caliban to run jobs that use your Linux machine's GPU. - Setting up a Google Cloud
account
so you can submit your code to Google's Cloud AI
Platform with
caliban cloud
.
In this section we will use Caliban to train an image classification network (implemented in TensorFlow) with Caliban. We will:
- Train a neural network on the local machine
- Increase the model's accuracy by changing the learning rate with a command-line flag
- Sweep across a range of learning rates with Caliban's experiment broadcasting feature
- Train the model in the Cloud on Google's AI Platform
- Develop your code interactively using
caliban shell
in the exact same environment.
Create an empty directory and use curl
to download a python
script
that trains a basic neural betwork.
mkdir demo && cd demo
curl --output mnist.py https://raw.githubusercontent.com/google/caliban/master/tutorials/basic/mnist.py
Create a file called requirements.txt
to declare tensorflow-cpu
as a dependency:
echo "tensorflow-cpu" > requirements.txt
Caliban will automatically make any entry in requirements.txt
available when
you run your code. See "Declaring
Requirements"
for more information.
Run this command to train your first ML model:
caliban run --nogpu mnist.py
You should see a stream of output ending in this:
Training model with learning rate=0.1 for 3 epochs.
Epoch 1/3
1875/1875 - 3s - loss: 2.0989 - accuracy: 0.2506
Epoch 2/3
1875/1875 - 3s - loss: 1.9222 - accuracy: 0.2273
Epoch 3/3
1875/1875 - 3s - loss: 2.0777 - accuracy: 0.1938
Model performance:
313/313 - 0s - loss: 2.0973 - accuracy: 0.1858
Your model was able to recognize digits from the MNIST dataset with 18.58% accuracy. Can we do better?
The default learning rate is 0.1
. Run the code again with a smaller learning
rate by passing a command-line flag, separated from your original command by
--
:
$ caliban run --nogpu mnist.py -- --learning_rate 0.01
<<elided>>
Training model with learning rate=0.01 for 3 epochs.
Epoch 1/3
1875/1875 - 4s - loss: 0.2676 - accuracy: 0.9221
Epoch 2/3
1875/1875 - 4s - loss: 0.1863 - accuracy: 0.9506
Epoch 3/3
1875/1875 - 4s - loss: 0.1567 - accuracy: 0.9585
Model performance:
313/313 - 0s - loss: 0.1410 - accuracy: 0.9642
96% accuracy! Much better! Can we do better still?
Caliban's experiment broadcasting feature will allow us to run many jobs with different sets of arguments.
Create a file called experiment.json
with a
JSON dictionary of the format
{"flag_name": ["list", "of", "values"]}
:
echo '{"learning_rate": [0.01, 0.001, 0.0001]}' > experiment.json
Pass the config with --experiment_config
and run again:
caliban run --experiment_config experiment.json --nogpu mnist.py
You should see accuracies of roughly 0.9493
, 0.9723
and 0.9537
. Looks like
0.001
is a nice choice.
Now it's time to submit the job to Cloud AI Platform.
(NOTE: This section requires a Google Cloud account. You can create a free account with $300 of credit to get started. Follow Caliban's "Getting Started with Google Cloud" documentation, then come back here to proceed.)
Submit the job to AI Platform by changing the word run
to cloud
:
caliban cloud --nogpu mnist.py -- --learning_rate 0.01
You should see output like this:
I0615 19:57:43.354172 4563361216 core.py:161] Job 1 - jobId: caliban_totoro_1, image: gcr.io/research-3141/974a776e6037:latest
I0615 19:57:43.354712 4563361216 core.py:161] Job 1 - Accelerator: {'count': 0, 'type': 'ACCELERATOR_TYPE_UNSPECIFIED'}, machine: 'n1-highcpu-32', region: 'us-central1'
I0615 19:57:43.355082 4563361216 core.py:161] Job 1 - Experiment arguments: ['--learning_rate', '0.01']
I0615 19:57:43.355440 4563361216 core.py:161] Job 1 - labels: {'gpu_enabled': 'false', 'tpu_enabled': 'false', 'job_name': 'caliban_totoro', 'learning_rate': '0_01'}
I0615 19:57:43.356621 4563361216 core.py:324] Submitting request!
I0615 19:57:45.078382 4563361216 core.py:97] Request for job 'caliban_totoro_20200615_195743_1' succeeded!
I0615 19:57:45.078989 4563361216 core.py:98] Job URL: https://console.cloud.google.com/ai-platform/jobs/caliban_totoro_20200615_195743_1?projectId=totoro-project
I0615 19:57:45.079524 4563361216 core.py:100] Streaming log CLI command: $ gcloud ai-platform jobs stream-logs caliban_totoro_20200615_195743_1
Submitting caliban_totoro_1: 100%|####################################################################################################################################################################################| 1/1 [00:02<00:00, 2.65s/requests]
I0615 19:57:45.405600 4563361216 core.py:673]
I0615 19:57:45.405819 4563361216 core.py:676] Visit https://console.cloud.google.com/ai-platform/jobs/?projectId=research-3141 to see the status of all jobs.
I0615 19:57:45.405959 4563361216 core.py:677]
This output means that Caliban has:
- built a Docker container with all of your code
- Pushed that container up to Google Cloud's Container Registry
- Submitted the job to AI Platform.
You can now visit the link in the output that looks like: https://console.cloud.google.com/ai-platform/jobs/caliban_totoro_20200615_195743_1?projectId=totoro-project to see all of your job's logs.
With Google Cloud, you can use on-demand GPUs and TPUs and train models on large datasets at very high speeds. You can also customize the machine type that AI Platform uses to run your job. You might need high memory or more CPU, for example.
See Caliban's "Customizing Machines and GPUs" for more information.
caliban shell
lets
you develop code interactively inside of the exact same environment that your
code will have available, locally during caliban run
or in the Cloud with
caliban cloud
.
Run the following command to activate the shell:
caliban shell --nogpu
You should see Caliban's terminal:
I0611 12:33:17.551121 4500135360 docker.py:911] Running command: docker run --ipc host -w /usr/app -u 735994:89939 -v /Users/totoro/code/example:/usr/app -it --entrypoint /bin/bash -v /Users/totoro:/home/totoro ab8a7d7db868
_________ __ ________ ___ _ __ __ __
/ ____/ | / / / _/ __ )/ | / | / / \ \ \ \
/ / / /| | / / / // __ / /| | / |/ / \ \ \ \
/ /___/ ___ |/ /____/ // /_/ / ___ |/ /| / / / / /
\____/_/ |_/_____/___/_____/_/ |_/_/ |_/ /_/ /_/
You are running caliban shell as user with ID 735994 and group 89939,
which should map to the ID and group for your user on the Docker host. Great!
[totoro@6a9b28990757 /usr/app]$
You're now living in an isolated Docker
container with your
tensorflow-cpu
dependency available (and any others you've
declared).
Run the python
command and check that tensorflow
is installed:
$ python
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'2.2.0'
Your home directory and the folder where you ran the command are both mounted into this isolated environment, so any changes you make to either of those directories will be reflected immediately.
Any code you add to the current folder and edit on your computer will be available in this special Caliban shell. Run the example from before like this:
python mnist.py --learning_rate 0.01
If your code runs in caliban shell
, you can be almost certain that your code
will execute in a Cloud environment, with potentially many GPUs attached and
much larger machines available.
Read the Overview for more information on Caliban's subcommands, then head over to Caliban's documentation site and check out the links on the sidebar.
If you find anything confusing, please feel free to create an issue on our Github Issues page, and we'll get you sorted out.
Caliban provides five subcommands that you run inside some project directory on your machine:
-
caliban shell
generates a Docker image containing any dependencies you've declared in arequirements.txt
and/orsetup.py
in the directory and opens an interactive shell in that directory. Thecaliban shell
environment is ~identical to the environment that will be available to your code when you submit it to AI Platform; the difference is that your current directory is live-mounted into the container, so you can develop interactively. -
caliban notebook
starts a Jupyter notebook or lab instance inside of a Docker image containing your dependencies; the guarantee about an environment identical to AI Platform applies here as well. -
caliban run
packages your directory's code into the Docker image and executes it locally usingdocker run
. If you have a GPU, the instance will attach to it by default - no need to install the CUDA toolkit. The Docker environment takes care of all that. This environment is truly identical to the AI Platform environment. The Docker image that runs locally is the same image that will run in AI Platform. -
caliban cloud
allows you to submit jobs to AI Platform that will run inside the same Docker image you used withcaliban run
. You can submit hundreds of jobs at once. Any machine type, GPU count, and GPU type combination you specify will be validated client side, so you'll see an immediate error with suggestions, rather than having to debug by submitting jobs over and over. -
caliban build
builds the Docker image used incaliban cloud
andcaliban run
without actually running the container or submitting any code. -
caliban cluster
creates GKE clusters and submits jobs to GKE clusters. -
caliban status
displays information about all jobs submitted by Caliban, and makes it easy to interact with large groups of experiments. Usecaliban status
when need to cancel pending jobs, or re-build a container and resubmit a batch of experiments after fixing a bug.
This is a research project, not an official Google product. Expect bugs and sharp edges. Please help by trying out Caliban, reporting bugs, and letting us know what you think!
Please refer to our Contributor's Guide for information on how to get started contributing to Caliban.
If Caliban helps you in your research, please consider citing the repository:
@software{caliban2020github,
author = {Vinay Ramasesh and Sam Ritchie and Ambrose Slone},
title = {{Caliban}: Docker-based job manager for reproducible workflows},
url = {http:https://github.com/google/caliban},
version = {0.1.0},
year = {2020},
}
In the above bibtex entry, names are in alphabetical order, the version number is intended to be that of the latest tag on github, and the year corresponds to the project's open-source release.
Copyright 2020 Google LLC.
Licensed under the Apache License, Version 2.0.