tutorial_conda_travis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ensuring reproducibility and transparency can be so easy when using the right tooling from the start. This is a tutorial to showcase the use of [conda](https://conda.io) and [Travis-CI](https://docs.travis-ci.com/). We use both to ensure easy reproducibility of our example notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```console\n",
    "foo@bar:~$ git clone https://github.com/HumanCapitalAnalysis/student-project-template\n",
    "foo@bar:~$ cd student-project-template\n",
    "foo@bar:~$ conda env create -f environment.yml\n",
    "foo@bar:~$ conda activate student_project\n",
    "foo@bar:~$ jupyter nbconvert --execute student_project.ipynb \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## conda - package management"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: conda [-h] [-V] command ...\n",
      "\n",
      "conda is a tool for managing and deploying applications, environments and packages.\n",
      "\n",
      "Options:\n",
      "\n",
      "positional arguments:\n",
      "  command\n",
      "    clean        Remove unused packages and caches.\n",
      "    config       Modify configuration values in .condarc. This is modeled\n",
      "                 after the git config command. Writes to the user .condarc\n",
      "                 file (/home/peisenha/.condarc) by default.\n",
      "    create       Create a new conda environment from a list of specified\n",
      "                 packages.\n",
      "    help         Displays a list of available conda commands and their help\n",
      "                 strings.\n",
      "    info         Display information about current conda install.\n",
      "    init         Initialize conda for shell interaction. [Experimental]\n",
      "    install      Installs a list of packages into a specified conda\n",
      "                 environment.\n",
      "    list         List linked packages in a conda environment.\n",
      "    package      Low-level conda package utility. (EXPERIMENTAL)\n",
      "    remove       Remove a list of packages from a specified conda environment.\n",
      "    uninstall    Alias for conda remove.\n",
      "    run          Run an executable in a conda environment. [Experimental]\n",
      "    search       Search for packages and display associated information. The\n",
      "                 input is a MatchSpec, a query language for conda packages.\n",
      "                 See examples below.\n",
      "    update       Updates conda packages to the latest compatible version.\n",
      "    upgrade      Alias for conda update.\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help     Show this help message and exit.\n",
      "  -V, --version  Show the conda version number and exit.\n",
      "\n",
      "conda commands available from other packages:\n",
      "  build\n",
      "  convert\n",
      "  develop\n",
      "  env\n",
      "  index\n",
      "  inspect\n",
      "  metapackage\n",
      "  render\n",
      "  server\n",
      "  skeleton\n"
     ]
    }
   ],
   "source": [
    "! conda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conda 4.6.14\n"
     ]
    }
   ],
   "source": [
    "! conda --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a virtual environment for our student project and install some basic packages right from the beginning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING conda.base.context:use_only_tar_bz2(632): Conda is constrained to only using the old .tar.bz2 file format because you have conda-build installed, and it is <3.18.3.  Update or remove conda-build to get smaller downloads and faster extractions.\n",
      "Collecting package metadata (repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/peisenha/.local/share/anaconda3/envs/student_project_template\n",
      "\n",
      "  added / updated specs:\n",
      "    - numpy\n",
      "    - pandas\n",
      "\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  blas               pkgs/main/linux-64::blas-1.0-mkl\n",
      "  ca-certificates    pkgs/main/linux-64::ca-certificates-2019.5.15-0\n",
      "  certifi            pkgs/main/linux-64::certifi-2019.6.16-py37_0\n",
      "  intel-openmp       pkgs/main/linux-64::intel-openmp-2019.4-243\n",
      "  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0\n",
      "  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4\n",
      "  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0\n",
      "  libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0\n",
      "  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0\n",
      "  mkl                pkgs/main/linux-64::mkl-2019.4-243\n",
      "  mkl_fft            pkgs/main/linux-64::mkl_fft-1.0.12-py37ha843d7b_0\n",
      "  mkl_random         pkgs/main/linux-64::mkl_random-1.0.2-py37hd81dba3_0\n",
      "  ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1\n",
      "  numpy              pkgs/main/linux-64::numpy-1.16.4-py37h7e9f1db_0\n",
      "  numpy-base         pkgs/main/linux-64::numpy-base-1.16.4-py37hde5b4d6_0\n",
      "  openssl            pkgs/main/linux-64::openssl-1.1.1c-h7b6447c_1\n",
      "  pandas             pkgs/main/linux-64::pandas-0.24.2-py37he6710b0_0\n",
      "  pip                pkgs/main/linux-64::pip-19.1.1-py37_0\n",
      "  python             pkgs/main/linux-64::python-3.7.3-h0371630_0\n",
      "  python-dateutil    pkgs/main/linux-64::python-dateutil-2.8.0-py37_0\n",
      "  pytz               pkgs/main/noarch::pytz-2019.1-py_0\n",
      "  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5\n",
      "  setuptools         pkgs/main/linux-64::setuptools-41.0.1-py37_0\n",
      "  six                pkgs/main/linux-64::six-1.12.0-py37_0\n",
      "  sqlite             pkgs/main/linux-64::sqlite-3.28.0-h7b6447c_0\n",
      "  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0\n",
      "  wheel              pkgs/main/linux-64::wheel-0.33.4-py37_0\n",
      "  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4\n",
      "  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3\n",
      "\n",
      "\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "#\n",
      "# To activate this environment, use\n",
      "#\n",
      "#     $ conda activate student_project_template\n",
      "#\n",
      "# To deactivate an active environment, use\n",
      "#\n",
      "#     $ conda deactivate\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! conda env remove --name student_project_template\n",
    "! conda create -y --name student_project_template numpy pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can have a look at all environments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# conda environments:\n",
      "#\n",
      "base                     /home/peisenha/.local/share/anaconda3\n",
      "altruism_replication     /home/peisenha/.local/share/anaconda3/envs/altruism_replication\n",
      "copulpy                  /home/peisenha/.local/share/anaconda3/envs/copulpy\n",
      "dev_norpy                /home/peisenha/.local/share/anaconda3/envs/dev_norpy\n",
      "dev_respy                /home/peisenha/.local/share/anaconda3/envs/dev_respy\n",
      "dev_soepy                /home/peisenha/.local/share/anaconda3/envs/dev_soepy\n",
      "dev_trempy               /home/peisenha/.local/share/anaconda3/envs/dev_trempy\n",
      "estimagic                /home/peisenha/.local/share/anaconda3/envs/estimagic\n",
      "norpy                    /home/peisenha/.local/share/anaconda3/envs/norpy\n",
      "option_value             /home/peisenha/.local/share/anaconda3/envs/option_value\n",
      "ose_tutorials            /home/peisenha/.local/share/anaconda3/envs/ose_tutorials\n",
      "ose_utils                /home/peisenha/.local/share/anaconda3/envs/ose_utils\n",
      "ov_analysis              /home/peisenha/.local/share/anaconda3/envs/ov_analysis\n",
      "respy                    /home/peisenha/.local/share/anaconda3/envs/respy\n",
      "respy_estimagic          /home/peisenha/.local/share/anaconda3/envs/respy_estimagic\n",
      "respy_smm                /home/peisenha/.local/share/anaconda3/envs/respy_smm\n",
      "robupy                   /home/peisenha/.local/share/anaconda3/envs/robupy\n",
      "ruspy                    /home/peisenha/.local/share/anaconda3/envs/ruspy\n",
      "scientific_computing     /home/peisenha/.local/share/anaconda3/envs/scientific_computing\n",
      "soepy                    /home/peisenha/.local/share/anaconda3/envs/soepy\n",
      "student_project       *  /home/peisenha/.local/share/anaconda3/envs/student_project\n",
      "student_project_template     /home/peisenha/.local/share/anaconda3/envs/student_project_template\n",
      "teaching_labor           /home/peisenha/.local/share/anaconda3/envs/teaching_labor\n",
      "teaching_microeconometrics     /home/peisenha/.local/share/anaconda3/envs/teaching_microeconometrics\n",
      "teaching_microeconometrics_sandbox     /home/peisenha/.local/share/anaconda3/envs/teaching_microeconometrics_sandbox\n",
      "teaching_sandbox         /home/peisenha/.local/share/anaconda3/envs/teaching_sandbox\n",
      "trempy                   /home/peisenha/.local/share/anaconda3/envs/trempy\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! conda env list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now switch to the terminal window or the Anaconda prompt to activate the environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```console\n",
    "foo@bar:~$ conda acivate student_project_template\n",
    "foo@bar:~$ which python \n",
    "foo@bar:~$ conda list\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are free to add / and remove packages form the environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```console\n",
    "foo@bar:~$ conda install scipy\n",
    "foo@bar:~$ conda remove pandas\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only the installed packages will be available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```console\n",
    "foo@bar:~$ python -c \"import scipy\"\n",
    "foo@bar:~$ python -c \"import pandas\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Returning to the notebook we can automate the process of environment generation using **environment.yml** files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: student_project_template\n",
      "\n",
      "dependencies:\n",
      "- numpy\n",
      "- pandas\n",
      "- scipy \n"
     ]
    }
   ],
   "source": [
    "! cat environment_tutorial.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Remove all packages in environment /home/peisenha/.local/share/anaconda3/envs/student_project_template:\n",
      "\n",
      "Collecting package metadata (repodata.json): done\n",
      "Solving environment: done\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "#\n",
      "# To activate this environment, use\n",
      "#\n",
      "#     $ conda activate student_project_template\n",
      "#\n",
      "# To deactivate an active environment, use\n",
      "#\n",
      "#     $ conda deactivate\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! conda env remove --name student_project_template\n",
    "! conda env create -f environment_tutorial.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Travis-CI - continuous integration\n",
    "\n",
    "Now that we have automated the installation of all required software, we can move it to the cloud and execute our analysis there to ensure that there are no local dependencies that we are missing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have linked your projects to **Travis_CI** already, so all your commits are monitored now and since you have a **.travis.yml** in your repo a build will be triggered based on the instructions in there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "language: python\n",
      "\n",
      "python:\n",
      "  - 3.6\n",
      "\n",
      "bundler_args: --retry 3\n",
      "\n",
      "install:\n",
      "  - sudo apt-get update\n",
      "  - ./utils/travis_miniconda.sh\n",
      "  - PATH=\"$HOME/miniconda/bin:$PATH\"\n",
      "  - conda env create -f environment.yml\n",
      "  - source activate student_project\n",
      "\n",
      "script:\n",
      "  - travis_wait python utils/travis_runner.py\n"
     ]
    }
   ],
   "source": [
    "! cat .travis.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of its content is boilerplate code from the [conda website](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/use-conda-with-travis-ci.html) that I moved to an [extra script](https://github.com/HumanCapitalAnalysis/student-project-template/blob/master/utils/travis_miniconda.sh). Now we can run the notebooks on the CI server to ensure full reproducibility as (if passing) this means that all required files are available on **GitHub** and the whole software environment is also fully specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/usr/bin/env python\n",
      "\"\"\"This script manages all tasks for the TRAVIS build server.\"\"\"\n",
      "import subprocess as sp\n",
      "import glob\n",
      "\n",
      "if __name__ == '__main__':\n",
      "\n",
      "    for notebook in glob.glob('*.ipynb'):\n",
      "        cmd = ' jupyter nbconvert --execute {}  --ExecutePreprocessor.timeout=-1'.format(notebook)\n",
      "        sp.check_call(cmd, shell=True)\n"
     ]
    }
   ],
   "source": [
    "! cat utils/travis_runner.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's trigger a build and see the magic in action."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](material/travis-ci.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When all is working, don't forget to proudly add your [badge](https://docs.travis-ci.com/user/status-images/) to the **README.md** file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}