This role deploys the HGVS variant nomenclature checker Mutalyzer.
New here? Check out How to use the Mutalyzer Ansible role.
- Debian 8 (Jessie) with a configured MTA
- Ansible version 2.0.1
The deployment uses the following tools:
- nginx as reverse proxy for Gunicorn and for serving static files.
- systemd for process control.
- Gunicorn as WSGI HTTP server.
- virtualenv for isolation of the Mutalyzer package and its Python dependencies.
- PostgreSQL for the database (can be customized).
- Redis for stat counters and other in-memory stores (can be customized).
Three applications are served by nginx:
- Website:
https://<servername>/
- HTTP/RPC+JSON webservice:
https://<servername>/json
- SOAP webservice:
https://<servername>/services
Logging is done at several levels in this stack:
- Mutalyzer log:
/opt/mutalyzer/log/mutalyzer.log
- Mutalyzer website Gunicorn log:
/opt/mutalyzer/versions/*/log/website.log
- Mutalyzer HTTP/RPC+JSON webservice Gunicorn log:
/opt/mutalyzer/versions/*/log/service-json.log
- Mutalyzer SOAP webservice Gunicorn log:
/opt/mutalyzer/versions/*/log/service-soap.log
- nginx access and error logs:
/var/log/nginx/
- redis server logs:
/var/log/redis/
- PostgreSQL server logs:
/var/log/postgresql/
Tool configurations can be found here (but you should never manually touch them):
- Mutalyzer configuration:
/opt/mutalyzer/versions/*/conf/settings.py
- Mutalyzer website Gunicorn configuration:
/opt/mutalyzer/versions/*/conf/website.conf
- Mutalyzer HTTP/RPC+JSON webservice Gunicorn configuration:
/opt/mutalyzer/versions/*/conf/service-json.conf
- Mutalyzer SOAP webservice Gunicorn configuration:
/opt/mutalyzer/versions/*/conf/service-soap.conf
- nginx configuration:
/etc/nginx/sites-available/mutalyzer-*
All Mutalyzer processes run as user mutalyzer
, which is also the owner of
everything under /opt/mutalyzer
. Some other Mutalyzer related locations are:
- Mutalyzer cache:
/opt/mutalyzer/cache/
- Mutalyzer Git clone:
/opt/mutalyzer/src/mutalyzer/
- Mutalyzer versions:
/opt/mutalyzer/versions/
In order to obtain zero-downtime deployments, we fix and isolate deployment versions. A version is identified by its Git commit hash and contains a Python virtual environment, configuration files, log files, and unix sockets for the website and webservices. Several versions can co-exist, but only one version is active (published).
The deployment of a new version is done when the Mutalyzer Git repository checkout changes:
- Create the new version in
/opt/mutalyzer/versions/<git commit hash>/
. - Run unit tests (if enabled, see variables below).
- Run database migrations.
- Start the new Gunicorn website and services.
- Test the availability of the website and services.
- Stop running batch processor and start a new one.
- Reload nginx with the new Gunicorn upstreams.
- Stop the old Gunicorn website and services (first completing all their pending requests).
In step 7, nginx will complete all existing requests from the old configuration while accepting requests with the new configuration, so this is zero-downtime.
The order above also means database migrations must always keep compatibility with the existing codebase, so some may have to be broken down into several steps and completed over several deployments.
Migrations that should be completed over several deployments are grouped per
Mutalyzer release. It is therefore advised to always update Mutalyzer one
release at a time and never skip a release to ensure database consistency and
zero-downtime deployments. This can be controled by setting the
mutalyzer_git_branch
role variable to a release tag.
The ~/.bashrc
file for user mutalyzer
activates the Mutalyzer Python
virtual environment and sets the MUTALYZER_SETTINGS
environment
variable. Administrative work is best done as that user:
sudo -u mutalyzer -i
Please note that you should re-source ~/.bashrc
in any existing shell
sessions after deploying a new Mutalyzer version, to switch to the current virtual
environment.
This role depends on the following roles:
https://git.lumc.nl/humgen-devops/ansible-role-postgresql
Variable overrides:
postgresql_databases:
- name: mutalyzer
encoding: UTF8
lc_collate: 'en_US.UTF-8'
lc_ctype: 'en_US.UTF-8'
backup: true
postgresql_users:
- name: mutalyzer
password: "{{ mutalyzer_database_password }}"
attributes: NOSUPERUSER,NOCREATEDB
databases:
- name: mutalyzer
privileges: ALL
This role is only needed when the mutalyzer_database_url
variable is null
(default).
https://git.lumc.nl/humgen-devops/ansible-role-redis
This role is only needed when the mutalyzer_redis_url
variable is null
(default).
https://git.lumc.nl/humgen-devops/ansible-role-nginx
https://git.lumc.nl/humgen-devops/ansible-role-mail-service-status
Also see variables of dependencies.
Default: localhost-insecure.crt
(self-signed certificate for localhost
)
SSL certificate file.
Default: localhost-insecure.key
SSL certificate keyfile.
Default: null
URL to use for connecting to a database in a form accepted by SQLAlchemy (see SQLAlchemy Database Urls).
If null
, a local PostgreSQL database is used (managed by this role). If a
database server is specified, it should be running when applying this role.
Default: insecure_password
Password for the database user. Only used when mutalyzer_database_url
is
null
(the password is part of the URL otherwise).
Default: null
URL to use for connecting to a Redis server in a form accepted by redis-py (see redis-py documentation).
If null
, a local Redis server is used (managed by this role). If another
Redis server is specified, it should be running when applying this role.
Default: localhost
Server name by which Mutalyzer can be reached.
Default: 60
Nginx read timeout for the website Gunicorn upstream.
Default: sync
Type of Gunicorn worker for the website. Must be one of sync
, eventlet
,
gevent
.
Default: 2
Number of Gunicorn workers for the website.
Default: 30
Timeout before killing silent Gunicorn workers for the website.
Default: 60
Nginx read timeout for the HTTP/RPC+JSON webservice Gunicorn upstream.
Default: sync
Type of Gunicorn worker for the HTTP/RPC+JSON webservice. Must be one of
sync
, eventlet
, gevent
.
Default: 1
Number of Gunicorn workers for the HTTP/RPC+JSON webservice.
Default: 30
Timeout before killing silent Gunicorn workers for the HTTP/RPC+JSON webservice.
Default: 60
Nginx read timeout for the SOAP webservice Gunicorn upstream.
Default: sync
Type of Gunicorn worker for the SOAP webservice. Must be one of sync
,
eventlet
, gevent
.
Default: 1
Number of Gunicorn workers for the SOAP webservice.
Default: 30
Timeout before killing silent Gunicorn workers for the SOAP webservice.
Default: noreply@localhost
Address used as sender in batch job notification emails.
Default: 52428800 (50 MB)
Maximum size of the cache directory (in bytes).
Default: 50000 (50 Kbp)
Maximum sequence length for the description extractor (in bases).
Default: []
List of remotes to sync the cache with, where each remote is a dictionary with the following fields:
name
: Remote name (should contain only letters, digits,-
, and_
).wsdl
: Location of the remote WSDL description.url_template
: URL for remote downloads, in which the filename is to be substituted for{file}
.
Default: null
Piwik web analytics configuration as a dictionary with the following fields:
base_url
: Piwik server base URL (include protocol, no trailing slash).site_id
: Piwik site ID.
Default: true
Whether or not to remove old Mutalyzer versions, including their Python virtual environment, log files, and configuration.
Default: false
Whether or not to run Mutalyzer unit tests. Deployment will be aborted if they fail.
Default: https://github.com/mutalyzer/mutalyzer.git
Mutalyzer Git repository URL to clone.
Default: master
Mutalyzer Git repository branch to checkout.
Easy deployment on a local virtual machine using Vagrant is provided in the vagrant directory.