pythoncourse/installation/installing_anaconda_and_pycharm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# Installing Anaconda and PyCharm\n",
    "\n",
    "Saeed Amen - Copyright Cuemacro 2020 - https://www.cuemacro.com - saeed@cuemacro.com\n",
    "\n",
    "In this notebook, we discuss how to install the Anaconda distribution of Python and other applications you need to run do development in Python, which will be useful for doing financial analysis in Python with libraries including the Cuemacro libraries, finmarketpy, findatapy and chartpy. \n",
    "\n",
    "You can also download a lot of this material from https://github.com/cuemacro/teaching - in particular the scripts for installing the conda environments. We'll also be installing a lot of other useful libraries for machine learning, natural language processing etc. \n",
    "\n",
    "We'll also make other suggestions so you can do financial data analysis, such as making sure that your firewalls allow access to sites like Quandl."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make sure firewalls are open to allow access to market data\n",
    "\n",
    "This is an important one! Make sure that you can download data via Python for libraries like Quandl and also to install Python. \n",
    "\n",
    "* Once you've done your Python installation (and have signed up for a Quandl account and API key for free), try running this code below to check. \n",
    "* Your firewall should also allow you to have access to allow the downloading of packages via conda and pip (if it isn't then you can't do much else, although in some circumstances you might be able to change the pip download URL to a site that does work)\n",
    "* If opening up firewalls is too much of an issue or you don't have admin rights to install Python locally, then one easier way to get started with Python for learning purposes, is to use something Azure Notebooks (see below). \n",
    "* This will however, at the very least need the website https://colab.research.google.com/ to be accessible or website https://repl.it/languages/python3 which offers online Python execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quandl.ApiConfig.api_key = \"TYPE YOUR KEY HERE\"\n",
    "\n",
    "# Download data for WTI Crude oil from the EIA database on Quandl (US Department of Energy)\n",
    "# We have looked up this ticker earlier on Quandl's website\n",
    "df_wti = pd.DataFrame(quandl.get(\"EIA/PET_RWTC_D\", start_date=\"2010-01-01\", end_date=\"2019-12-31\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Download and install Anaconda\n",
    "\n",
    "You can download install the Anaconda distribution for Python from https://www.anaconda.com/distribution \n",
    "\n",
    "There are versions of Anaconda for Windows, Linux and Mac operating systems. *Make sure you install the 64 bit version of Anaconda*. The 32 bit version is not compatible with some of the Python libraries we'll be using, and furthermore, it'll likely run out of memory easily.\n",
    "\n",
    "By default the Anaconda will be installed at the following folder, which will depend on your username. Make sure you keep a note of where you installed Anaconda for later, in particular when you need to tell Anaconda where you installed the distribution!\n",
    "\n",
    "* Windows: `C:\\Users\\<your-username>\\Anaconda3\\`\n",
    "* macOS: `/Users/<your-username>/anaconda3`\n",
    "* Linux: `/home/<your-username>/anaconda3`\n",
    "\n",
    "Anaconda comes with many data science focused Python libraries. However, we'll still need to install quite a few other extra ones. Also in some instances, we'll install a different versions of certain libraries (including an earlier one for Pandas).\n",
    "\n",
    "On Windows It is also recommended to add some Anaconda folders (should look similar to the below) to your Windows PATH (the Anaconda installer usually has a setting you can tick for this, but if that doesn't work do it in your environment variables in Control Panel). If this isn't set you can have issues when running certain libraries like xlwings.\n",
    "\n",
    "* `C:\\Users\\<your-username>\\Anaconda3`\n",
    "* `C:\\Users\\<your-username>\\Anaconda3\\Scripts`\n",
    "* `C:\\Users\\<your-username>\\Anaconda3\\Library\\bin`\n",
    "\n",
    "You might need to allow Win32 long paths, otherwise Windows will restrict the number of folders on your path, can end up being very long with Python installations! See https://stackoverflow.com/questions/26155135/node-npm-windows-file-paths-are-too-long-to-install-packages/37528731#37528731 on how to set this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Windows: Installing a conda environment for Windows from YML file (quickest method - recommended)\n",
    "\n",
    "We can download [environment_windows.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_windows.yml) from our browser or via `curl` in the command line. We can use that YML file to create our conda environment with all the necessary packages. This also installs the exact same libraries versions that I have (and reduces the likelihood of version conflicts between libraries). The conda environment file is periodically updated for new versions of libraries.\n",
    "\n",
    "* Open up the Anaconda Prompt (should be in the Start Menu)\n",
    "* In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in `conda activate` and press Enter to exit the current conda environment, then `conda remove -n py37class --all --yes` and press Enter to remove any existing environments called `py37class`\n",
    "    * If you haven't already downloaded the `environment_windows.yml` file you can do it from the command line using `curl`:\n",
    "        * Type `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_windows.yml > environment_windows.yml` and press Enter\n",
    "    * Type in `conda env create -f environment_windows.yml` and press Enter\n",
    "    * Anaconda will then run all the necessary command to download the various packages above using conda and pip\n",
    "    * At the end of the process you'll have a Python 3.7 conda environment with the name 'py37class' which you can use, with all the packages you'll need for this course, such as pandas, numpy etc.\n",
    "    * Note that you may need to put in the full path for wherever you downloaded the `environment_windows.yml` file or you can simply `cd` to that folder and then run the above conda command\n",
    "    * Typically when you download files from your internet browser, Windows will usually save it in the users downloads folder, this is usually accessable by doing\n",
    "        `cd \\Users\\<your-username>\\Downloads` and then pressing Enter\n",
    "    \n",
    "* The `environment_windows.yml` file (or similar name) basically has all the instructions required to recreate a conda environment\n",
    "* To create your own `environment.yml` file (for backup purposes, or if you'd like to distribute your conda environment) run the below command in your Anaconda Prompt\n",
    "    * `conda env export > environment_windows.yml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:00:00.009779Z",
     "start_time": "2019-10-01T16:00:00.000993Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Windows: Installing a conda environment for Windows (slower method)\n",
    "\n",
    "A conda environment is a separate version of Python, where you can install all your own Python environments. This is a slower way to do it, but this makes it easier to change the versions of the libraries (note, might not be 100% same versions as my libraries!)\n",
    "\n",
    "* Download \n",
    "* Open up Notepad\n",
    "* Copy the below text\n",
    "* Create a folder `C:\\pythoncourse` (or if you create it elsewhere, keep a note of that)\n",
    "* Save file `create_conda_env_windows.bat` (see download link below) in the `C:\\pythoncourse` folder\n",
    "* Open up the Anaconda Prompt (should be in the Start Menu)\n",
    "* In this prompt, your Anaconda folder will on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "* Type in\n",
    "    `cd\\`\n",
    "* This will change directory to the `C:\\` drive\n",
    "* Then run\n",
    "    `cd\\pythoncourse`\n",
    "* Then run\n",
    "    `create_conda_env_windows.bat`\n",
    "* It will likely take a long time, and you might need to periodically click yes in Windows to allower the installer to change your settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### [Download create_conda_env_windows.bat from GitHub](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_windows.bat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excel: Installing xlwings addin in Excel on Windows (or Mac)\n",
    "\n",
    "Both of the above should install the xlwings Python library. You also need to add the xlwings addin to Excel too. Instructions for this can be found at https://docs.xlwings.org/en/stable/addin.html (xlwings is also supported on Excel for Mac, although the functionality may differ). It won't work on Linux, given there's no Linux version of Excel). Usually this will involve, closing Excel, then running the following commands in your Anaconda Prompt, before restarting Excel:\n",
    "\n",
    "    conda activate py37class\n",
    "    xlwings addin install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linux/Mac: Installing a conda environment for Linux/Mac from YAML file (quickest method - recommended)\n",
    "\n",
    "We can download [environment_linux.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_linux.yml) if we're installing on Linux or [environment_mac.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_mac.yml) for Mac from our browser or via `curl` in the command line. We can use that YML file to create our conda environment with all the necessary packages . This should be a lot faster to run. This also installs the exact same libraries versions that I have (and reduces the likelihood of version conflicts between libraries). The conda environment file is periodically updated for new versions of libraries.\n",
    "\n",
    "* Open up a terminal window (making sure that Anaconda is on your path) on Linux or Mac\n",
    "* In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in `conda activate` to exit the current conda environment and press Enter, then `conda remove -n py37class --all --yes` and press Enter to remove any existing environments called `py37class`\n",
    "    * If you haven't already downloaded the `environment_linux.yml` or `environment_mac.yml` file you can do it from the command line using `curl`:\n",
    "        * Type `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_linux.yml > environment_linux.yml` and press Enter (or `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_mac.yml > environment_mac.yml` on Mac)\n",
    "    * Type in `conda env create -f environment_linux.yml` and press Enter (or `conda env create -f environment_mac.yml` on Mac)\n",
    "    * Anaconda will then run all the necessary command to download the various packages above using conda and pip\n",
    "    * At the end of the process you'll have a Python 3.6 conda environment with the name 'py37class' which you can use, with all the packages you'll need for this course, such as pandas, numpy etc.\n",
    "    * Note that you may need to put in the full path for wherever you downloaded the `environment_linux.yml` (or `environment_mac.yml`) file or you can simply `cd` to that folder and then run the above conda command\n",
    "    * Typically on Mac OS X when you save down files from your internet browser they will be in the user's home downloads folder, which you can get to by typing\n",
    "        `cd ~/Downloads` and then pressing Enter\n",
    "    \n",
    "* The `environment_linux.yml` file (or `environment_mac.yml` or similar name) basically has all the instructions required to recreate a conda environment\n",
    "* To create your own `environment.yml` file (for backup purposes, or if you'd like to distribute your conda environment) run the below command in your Anaconda Prompt\n",
    "    * `conda env export > environment_linux.yml` on Linux\n",
    "    * `conda env export > environment_mac.yml` on Mac"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:12:06.412884Z",
     "start_time": "2019-10-01T16:12:06.404503Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Linux/Mac: Installing a conda environment for Linux/Mac (slower method)\n",
    "\n",
    "A conda environment is a separate version of Python, where you can install all your own Python environments. For Linux \n",
    "and Mac, we'll install more libraries, which you might need to use later (some of which aren't fully supported by Windows).\n",
    "\n",
    "* Download \n",
    "* Open up a text editor\n",
    "* Copy the below text\n",
    "* Create a folder `/Users/<your-username>/pythoncourse` for Mac or `/home/<your-username>/pythoncourse` for Linux\n",
    "* Save file `create_conda_env_linux.sh` or `create_conda_env_mac.sh` (see download link below) in the pythoncourse folder\n",
    "* Open a Terminal window (usually a black window icon on both Linux and Mac)\n",
    "* Type in (or wherever you installed Anaconda) and press enter to run\n",
    "    * `cd /Users/<your-username>/anaconda3/bin` for Mac\n",
    "    * `cd /home/<your-username>/anaconda3/bin` for Linux\n",
    "* conda is in this folder\n",
    "* Note, conda might already be on your path - you can check this by typing `conda` and pressing enter, to see if you get an error. On Windows\n",
    "the installer will have a default option to add Anaconda to your path\n",
    "* On Linux/Mac, it is recommend to add the Anaconda folder to your path\n",
    "* You can temporarily add paths in Linux/Mac using the following command in the terminal (change the path to wherever you installed\n",
    "Anaconda)\n",
    "    * `export PATH=/Users/<your-username>/anaconda3/bin:$PATH` for Mac\n",
    "    * `export PATH=/home/<your-username>/anaconda3/bin:$PATH` for Linux\n",
    "* To permanently add folders to your path by editing a file like `.bashrc` which is in your home folder\n",
    "    * http:https://osxdaily.com/2015/07/28/set-enviornment-variables-mac-os-x/ for Mac\n",
    "    * https://opensource.com/article/17/6/set-path-linux for Linux\n",
    "* Then run\n",
    "    * `./Users/<your-username>/pythoncourse/create_conda_env_mac.sh` for Mac\n",
    "    * `./home/<your-username>/pythoncourse/create_conda_env_linux.sh` for Linux\n",
    "* It will likely take a long time, and you might need to periodically click yes also Windows to change settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:13:50.842110Z",
     "start_time": "2019-10-01T16:13:50.828861Z"
    },
    "pycharm": {}
   },
   "source": [
    "### [Download create_conda_env_linux.sh from GitHub for Linux](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_linux.sh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Download create_conda_env_mac.sh from GitHub for Mac](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_mac.sh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:04:01.263522Z",
     "start_time": "2019-10-01T16:04:01.258640Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Download and install PyCharm\n",
    "\n",
    "* If you running Python locally, you can try to install PyCharm, which makes it easier to develop Python code\n",
    "* You can download and install PyCharm Community from https://www.jetbrains.com/pycharm/download\n",
    "* There are versions for PyCharm for Windows, Linux and Mac operating systems.\n",
    "* PyCharm IDE makes it easier to write and run Python code\n",
    "* In File / Settings - you will likely need to set PyCharm to use your py37class environment, which is likely at the following locations\n",
    "    * Windows: `C:\\Users\\<your-username>\\Anaconda3\\envs\\py37class`\n",
    "    * macOS: `/Users/<your-username>/anaconda3/envs/py37class`\n",
    "    * Linux: `/home/<your-username>/anaconda3/envs/py37class`\n",
    "* PyCharm will create a workspace for you, where you can place your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Google Colab with Jupyter\n",
    "\n",
    "If you do not want to install Anaconda on your own machine (or if you don't have the correct permissions to do so), you can instead use Google Colab, which gives you a Jupyter notebook in the cloud. This can be a good solution for those wanting to learn Python. You'll need a Google account, otherwise you can't save down your notebooks. You can access it at https://colab.research.google.com/\n",
    "\n",
    "If you find there are libraries which aren't available, you can install these in Google Colab (as with any Jupyter notebook) using `!pip` inside the notebook. Below, we have a number of useful libraries to get you started. You might need to run more than these. Note that you might need to re-run pip on Google Colab every so often, because the server gets restarted, and the library installations will be lost. \n",
    "\n",
    "There are several other alternatives to Google Colab, such as https://cocalc.com/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \\\n",
    "  redis pathos pyarrow==2.0.0 pandas==1.0.5 quandl \\\n",
    "  finmarketpy chartpy findatapy \\\n",
    "  cufflinks==0.17.3 kaleido \\\n",
    "  plotly==4.9.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional installations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Download and install Git (optional)\n",
    "\n",
    "Git is version control software, which maybe useful to install some Python libraries we'll use (in practice you can install these without Git, but they might not be the latest versions). It's also worth understanding how to use version control if you want to code later! You can download and install Git for Windows, Linux and Mac operating systems from https://git-scm.com/downloads\n",
    "\n",
    "Note that for Linux, you can install from the command line but the syntax depends on your Linux distribution https://git-scm.com/download/linux discusses this in some detail.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "### GPU versions of TensorFlow and PyTorch (optional)\n",
    "\n",
    "If you want to use your GPU for certain operations in particular for PyTorch and TensorFlow, you may need to update your NVIDIA graphics driver. First check if you have a GPU accelerated graphics card which supports CUDA (usually most newer NVIDIA graphics cards). This means you can install GPU accelerated versions of machine libraries like TensorFlow and PyTorch. \n",
    "\n",
    "To do this, you need to manually install various CUDA libraries. For full details on how to install these see https://www.tensorflow.org/install/gpu (both for Windows and Linux). Once you've done that you can edit the scripts below where indicated, so the GPU enabled Python versions of PyTorch and TensorFlow are installed (rather than the CPU version). Note, the CPU versions work fine, but will be slower and by default the environments below install CPU versions for maximum compatibility in case you don't have an NVIDIA card. If you are running on the cloud, you need to check that the cloud machine you are using has a GPU. Typically free instances are CPU only."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Bloomberg to access data via API (optional)\n",
    "\n",
    "It is helpful to install Bloomberg's blpapi Python library if you can (and have a Bloomberg account and have a Bloomberg Terminal with Windows). \n",
    "\n",
    "The py37class environment for Windows already contains blpapi, so you don't need to do this step!\n",
    "\n",
    "However, if you want to install it without conda, you follow the instructions at https://github.com/cuemacro/findatapy/blob/master/BLOOMBERG.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing tabula-py and pytesseract (optional)\n",
    "\n",
    "If you'd like to use tabula-py (extracting tables from PDF) and pytesseract (for doing optical character recognition), as well as install the Python libraries with pip, you'll also need to some further steps:\n",
    "\n",
    "tabula-py uses the Java runtime underneath, hence it needs the Java runtime installed on your path.\n",
    "\n",
    "* Tabula installation on Windows - https://tabula-py.readthedocs.io/en/latest/getting_started.html#get-tabula-py-working-windows-10\n",
    "\n",
    "pytesseract is a wrapper for Tesseract, which needs to be installed first\n",
    "\n",
    "* Tesseract installation on Windows - https://github.com/UB-Mannheim/tesseract/wiki. \n",
    "* Tesseract installation on Linux at https://tesseract-ocr.github.io/tessdoc/Home.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test your Python installation by running this.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can test your Python installation, by starting the Anaconda prompt (switching to the right conda environment) and then starting Jupyter (in Azure Notebook, you'll start a Jupyter notebook via it's own interface) using the below commands. \n",
    "\n",
    "Note, that you will likely have to change the default notebook-dir parameter (or you can just omit it, in which case Jupyter will use the current working directory)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    conda activate py37class\n",
    "    jupyter notebook --notebook-dir='e:/cuemacro/pythoncourse/pythoncourse/notebooks'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then try running the below Python code in a Jupyter notebook or the Python interpreter to see if some of the libraries we've installed work. This is not an exhaustive test, but only a few which we'll use a lot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-04T18:38:51.129577Z",
     "start_time": "2019-12-04T18:38:51.123720Z"
    }
   },
   "outputs": [],
   "source": [
    "import chartpy\n",
    "import quandl\n",
    "import finmarketpy\n",
    "import findatapy\n",
    "import pandas\n",
    "import numpy\n",
    "import plotly"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "222.755px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}