This repository contains a heavily refactored copy of "Combine-Docker", a docker-compose setup for running Combine, an experimental / alpha-release cultural heritage metadata aggregation system.

Combine itself is a complicated system. The components:

• Combine Django app (combine-django): a django web application that serves up the primary GUIs (/admin and /combine);
• Combine Livy app (combine-livy): an instance of Livy used to manage Spark (below) and integrate (loosely) with Combine Django;
• Celery (combine-celery): an asynch job queue used by the Django app (and possibly Livy);
• Hadoop setup (combine-hadoop), comprised of one namenode and one datanode;
• Redis: a message broker for the Celery queue (and possibly used by Livy);
• MongoDB: a no-SQL document database, where records are stored;
• MySQL: the Django App database;
• and Elasticsearch: an index used by the Django app for facets and searching of jobs, records, etc.

Underlying a few of these components is an installation of Spark (using PySpark), as well as the DPLA Ingestion 3 Scala jars for running within Spark.

For this refactor, the following docker-compose services are defined, based off (where feasible) shipped & versioned Docker images:

• combine-django (built & shipped to dockerhub by us)
• combine-livy (built & shipped to dockerhub by us)
• combine-celery (built & shipped to dockerhub by us)
• hadoop-datanode (based on the image combine-hadoop, built & shipped to dockerhub by us)
• hadoop-namenode (based on the image combine-hadoop, built & shipped to dockerhub by us)
• redis (community image)
• mongo (community image)
• mysql (community image)
• elasticsearch (community image).

We also replaced the initialization and migration scripts with 2 additional docker-compose services that execute idempotent forms of the replaced commands:

• hadoop-namenode-format
• combine-django-migration

These services execute a command only, and run whenever docker-compose up or restart is run.

As a locally preferred wrapper to running the commands needed below for local development spin-up, testing, and deployment, we use a Makefile with the following scenarios:

• make build-combine-{enter image}: build the entered image locally, based on the directory of the referenced image.
• make tag={enter image tag} tag-combine-{enter image}: tag with the given tag & prepend tulibraries/ to the name of the referenced image.
• make tag={enter image tag} ship-combine-{enter image}: log into DockerHub (expects you've done this at least one locally) & ships the referenced image & tag to tulibraries dockerhub.
• make env-file={enter dotenv} up: cleans out the docker-compose combine-docker project directory, pulls the images, builds anything remaining, then runs docker-compose up.
• make reload: starts docker-compose services already running.
• make stop: stops, but does not delete, docker-compose services.
• make down: stops and rmoves all the docker-compose services, volumes, images spun up above.

WARNING: The containerization of Combine provides arguably easier deployment and upgrading, but introduces some risks to data integrity in Combine. Combine-Docker stores data that needs to persist between container rebuilding and upgrades in named volumes, specifically the following:

• esdata: ElasticSearch data
• mongodata: Mongo data
• mysqldata: MySQL data
• hdfs: Hadoop HDFS data
• combine_home: Home directory for Combine that contains important, non-volatile data

Ensuring that first_build.sh (or update_build.sh if appropriate) has been run, fire up all containers with the following:

docker-compose up -d

Logs can be viewed with the logs command, again, selecting all services, or a subset of:

# tail all logs
docker-compose logs -f

# tail logs of specific services
docker-compose logs -f combine-django combine-celery livy

As outlined in the Combine-Docker Containers section all services, or a subset of, can be restarted as follows:

# e.g. restart Combine Django app, background tasks Celery, and Livy
docker-compose restart combine-django combine-celery

# e.e. restart everything
docker-compose restart

To stop all services and containers (NOTE: Do not include -v or --volumes flags, as these will wipe ALL data from Combine):

docker-compose down

View stats of containers:

docker stats

This dockerized version of Combine supports, arguably, easier version updating becaues major components, broken out as images and containers, can be readily rebuilt. Much like the repository Combine-Playbook, this repository follows the same versioning as Combine. So checking out the tagged release v0.7 for this repository, will build Combine version v0.7.

To update, follow these steps from the Combine-Docker repository root folder:

# pull new changes to Combine-Docker repository
git pull

# checkout desired release, e.g. v0.7
git checkout v0.7

# run update build script
./update_build.sh

# Restart as per normal
docker-compose up -d

This dockerized version of Combine includes the following services, where each becomes a single container:

The following tables show Docker volumes and binds that are created to support data sharing between containers, and "long-term" data storage. The column Data Storage indicates which volumes act as data stores for Combine and should not be deleted (unless, of course, a fresh installation is desired). Conversely, the column Refreshed on Upgrade shows which tables are removed during builds. Note: this information is purely for informational purposes only; the build scripts and normal usage of docker-compose up and docker-compose down will not remove these volumes.

Depending on machine and OS (Linux, Mac, Windows), might need to bump vm.max_map_count on Docker host machine (seems to be particulary true on older ones): https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode

By default, nearly all relevant ports are exposed from the containers that conspire to run Combine, but these can turned off selectively (or changed) if you have services running on your host that conflict. Look for the ports section for each service in the docker-compose.yml to enable or disable them.

Please don't hesitate to submit an issue!

The Combine Django application, where most developments efforts are targeted, is a bind mount volume from the location of this cloned repository on disk at ./combine/combine. Though the application is copied to the docker images during build, to support the installation of dependencies, the location /opt/combine is overwritten by this bind volume at docker-compose up or run. This allows live editing of the local folder ./combine/combine, which is updating the folder /opt/combine in services combine-django, combine-celery, and livy.

The folder ./combine/combine can, for the most part, be treated like a normal GitHub repository. For example, one could checkout or create a new branch, and then push and pull from there.