pythoncourse/installation/installing_anaconda_and_pycharm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# Installing Anaconda and PyCharm\n",
    "\n",
    "Saeed Amen - Copyright Cuemacro 2021 - https://www.cuemacro.com - saeed@cuemacro.com\n",
    "\n",
    "In this notebook, we discuss how to install the Anaconda distribution of Python and other applications you need to run do development in Python, which will be useful for doing financial analysis in Python with libraries including the Cuemacro libraries, finmarketpy, findatapy and chartpy. \n",
    "\n",
    "You can also download a lot of this material from https://github.com/cuemacro/teaching - in particular the scripts for installing the conda environments. We'll also be installing a lot of other useful libraries for machine learning, natural language processing etc. \n",
    "\n",
    "We'll also make other suggestions so you can do financial data analysis, such as making sure that your firewalls allow access to sites like Quandl."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make sure firewalls are open to allow access to market data\n",
    "\n",
    "This is an important one! Make sure that you can download data via Python for libraries like Quandl and also to install Python. \n",
    "\n",
    "* Once you've done your Python installation and also the `py38class` environment detailed below (and have signed up for a Quandl account and API key for free), try running the code at the end of this guide to check it's installed the correct packages\n",
    "* Your firewall should also allow you to have access to allow the downloading of packages via conda and pip\n",
    "    * it isn't then you can't do much else, although in some circumstances you might be able to change the pip download URL to a site that does work/using proxy details provided by your firm, with\n",
    "    * [Using Anaconda behind a company proxy](https://docs.anaconda.com/anaconda/user-guide/tasks/proxy/)\n",
    "      * The below shows how we can configure conda to use proxy servers\n",
    "        * `conda config --set proxy_servers.http http:https://id:pw@proxyserver:port`\n",
    "        * `conda config --set proxy_servers.https https://id:pw@proxyserver:port`\n",
    "    * [Pip user guide](https://pip.pypa.io/en/stable/user_guide/)\n",
    "      * The below shows how we can specify the proxy server when downloading with pip\n",
    "        * `pip install --proxy=https://[id:pwd@]proxyserver:port somepackage`\n",
    "\n",
    "* If opening up firewalls is too much of an issue or you don't have admin rights to install Python locally, then one easier way to get started with Python for learning purposes, is to use something like Google Colab (see below). \n",
    "* This will however, at the very least need the website https://colab.research.google.com/ to be accessible (some firms may block some Google websites) or website https://repl.it/languages/python3 which offers online Python execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Download and install Anaconda\n",
    "\n",
    "You can download install the Anaconda distribution for Python from https://www.anaconda.com/distribution \n",
    "\n",
    "There are versions of Anaconda for Windows, Linux and Mac operating systems. *Make sure you install the 64 bit version of Anaconda*. **The 32 bit version is not compatible with some of the Python libraries we'll be using, and furthermore, it'll likely run out of memory easily.**\n",
    "\n",
    "By default the Anaconda will be installed at the following folder, which will depend on your username. Make sure you keep a note of where you installed Anaconda for later, in particular when you need to tell Anaconda where you installed the distribution!\n",
    "\n",
    "* Windows: `C:\\Users\\<your-username>\\Anaconda3\\`\n",
    "* macOS: `/Users/<your-username>/anaconda3` or ``/Users/<your-username>/opt/anaconda3`\n",
    "* Linux: `/home/<your-username>/anaconda3`\n",
    "\n",
    "Anaconda comes with many data science focused Python libraries. However, we'll still need to install quite a few other extra ones. Also in some instances, we'll install a different versions of certain libraries (including an earlier one for Pandas).\n",
    "\n",
    "On Windows It is also recommended to add some Anaconda folders (should look similar to the below) to your Windows PATH (the Anaconda installer usually has a setting you can tick for this, but if that doesn't work do it in your environment variables in Control Panel). If this isn't set you can have issues when running certain libraries like xlwings.\n",
    "\n",
    "* `C:\\Users\\<your-username>\\Anaconda3`\n",
    "* `C:\\Users\\<your-username>\\Anaconda3\\Scripts`\n",
    "* `C:\\Users\\<your-username>\\Anaconda3\\Library\\bin`\n",
    "\n",
    "You might need to allow Win32 long paths, otherwise Windows will restrict the number of folders on your path, can end up being very long with Python installations! See https://stackoverflow.com/questions/26155135/node-npm-windows-file-paths-are-too-long-to-install-packages/37528731#37528731 on how to set this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Windows: Installing a conda environment for Windows from YML file (quickest method - recommended)\n",
    "\n",
    "We can download [environment_windows.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_windows.yml) from our browser or via `curl` in the command line (or it will be alongside the teaching notes folder if you attend a Cuemacro course). We can use that YML file to create our conda environment with all the necessary packages. This also installs the exact same libraries versions that I have (and reduces the likelihood of version conflicts between libraries). The conda environment file is periodically updated for new versions of libraries. Sometimes conda might hang for a very long time, if this is the case, try the \"slower\" method below.\n",
    "\n",
    "* Open up the Anaconda Prompt (should be in the Start Menu usually labelled *Anaconda Prompt (Anaconda3)* try to avoid using *Anaconda Powershell Prompt (Anaconda3)*, because it can sometimes have issues with `curl`)\n",
    "* In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in `conda activate` and press Enter to exit the current conda environment, then `conda remove -n py38class --all --yes` and press Enter to remove any existing environments called `py38class`\n",
    "    * If you haven't already downloaded the `environment_windows.yml` file you can do it from the command line using `curl`:\n",
    "        * Type on one line `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_windows.yml > environment_windows.yml` and press Enter\n",
    "    * Type in `conda env create -f environment_windows.yml` and press Enter\n",
    "    * Note that if you get an error like `ruamel_yaml.reader.ReaderError: unacceptable character #x0000: special characters are not allowed in \"<unicode string>\", position 3` (see this GitHub [issue](https://github.com/conda/conda/issues/9749)) try running the below instead and press Enter in the Anaconda Prompt:\n",
    "        * `conda env create -f .\\environment_windows.yml`\n",
    "    * Anaconda will then run all the necessary command to download the various packages above using conda and pip\n",
    "    * Sometimes when you see the progress bar for certain libraries it will appear to stall, but you can sometimes try pressing \"y\" to make it continue\n",
    "    * At the end of the process you'll have a Python 3.8 conda environment with the name 'py38class' which you can use, with all the packages you'll need for this course, such as pandas, numpy etc.\n",
    "    * Note that you may need to put in the full path for wherever you downloaded the `environment_windows.yml` file or you can simply `cd` to that folder and then run the above conda command\n",
    "    * Typically when you download files from your internet browser, Windows will usually save it in the users downloads folder, this is usually accessable by doing\n",
    "        `cd \\Users\\<your-username>\\Downloads` and then pressing Enter\n",
    "    \n",
    "* The `environment_windows.yml` file (or similar name) basically has all the instructions required to recreate a conda environment\n",
    "* To create your own `environment.yml` file (for backup purposes, or if you'd like to distribute your conda environment) run the below command in your Anaconda Prompt\n",
    "    * `conda env export > environment_windows.yml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:00:00.009779Z",
     "start_time": "2019-10-01T16:00:00.000993Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Windows: Installing a conda environment for Windows (slower method)\n",
    "\n",
    "A conda environment is a separate version of Python, where you can install all your own Python environments. This is a slower way to do it, but this makes it easier to change the versions of the libraries (note, might not be 100% same versions as my libraries). Also try this method if the above one takes too long. \n",
    "\n",
    "Note that underneath `create_conda_env_windows.bat` are using [mamba](https://github.com/mamba-org/mamba) - which is a faster implementation of conda to install some libraries versus conda. If you have any difficulties with mamba, you might need to change references in `create_conda_env_windows.bat` where mamba is installing libraries to conda. If this method still doesn't work/or hangs, skip to the section \"Installing a conda environment if all else fails\".\n",
    "\n",
    "### Download `create_conda_env_windows.bat`\n",
    "* Open up the Anaconda Prompt (should be in the Start Menu)\n",
    "  * (Method 1) Download `create_conda_env_windows.bat`\n",
    "    * Open `create_conda_env_windows.bat` from GitHub by clicking this [link](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_windows.bat)\n",
    "    * Copy all the text from your browser\n",
    "    * Open up Notepad\n",
    "    * Paste the text\n",
    "    * Create a folder `C:\\pythoncourse` (or if you create it elsewhere, keep a note of that)\n",
    "    * Save file `create_conda_env_windows.bat` in the `C:\\pythoncourse` folder\n",
    "    * In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in\n",
    "        `cd\\`\n",
    "    * This will change directory to the `C:\\` drive\n",
    "    * Then run\n",
    "        `cd\\pythoncourse`\n",
    "  * (Method 2) Alternatively, you can download `create_conda_env_windows.bat` using the below command (typed all on one line in the Anaconda Prompt then press Enter:\n",
    "    * `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_windows.bat > create_conda_env_windows.bat`\n",
    "    \n",
    "### Kick off installation\n",
    "* Then type in the below in  in the Anaconda Prompt then press Enter:\n",
    "    `create_conda_env_windows.bat`\n",
    "* It will likely take a long time, and you might need to periodically click yes in Windows to allow the installer to change your settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excel: Installing xlwings addin in Excel on Windows (or Mac)\n",
    "\n",
    "Both of the above should install the xlwings Python library. You also need to add the xlwings addin to Excel too. Instructions for this can be found at https://docs.xlwings.org/en/stable/addin.html (xlwings is also supported on Excel for Mac, although the functionality may differ). It won't work on Linux, given there's no Linux version of Excel). Usually this will involve, closing Excel, then running the following commands in your Anaconda Prompt, before restarting Excel:\n",
    "\n",
    "    conda activate py38class\n",
    "    xlwings addin install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linux/Mac: Installing a conda environment for Linux/Mac from YAML file (quickest method - recommended)\n",
    "\n",
    "We can download [environment_linux.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_linux.yml) if we're installing on Linux or [environment_mac.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_mac.yml) for Mac from our browser or via `curl` in the command line. We can use that YML file to create our conda environment with all the necessary packages . This should be a lot faster to run. This also installs the exact same libraries versions that I have (and reduces the likelihood of version conflicts between libraries). The conda environment file is periodically updated for new versions of libraries. Sometimes conda might hang for a very long time, if this is the case, try the \"slower\" method below.\n",
    "\n",
    "* Open a Terminal window (usually a black window icon on both Linux and Mac) making sure that Anaconda is on your path on Linux or Mac\n",
    "  * See the section **Make sure Anaconda is on your Mac or Linux path** for more information about starting conda, if Anaconda isn't on your paht\n",
    "* In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in `conda activate` to exit the current conda environment and press Enter, then `conda remove -n py38class --all --yes` and press Enter to remove any existing environments called `py38class`\n",
    "    * If you haven't already downloaded the `environment_linux.yml` or `environment_mac.yml` file you can do it from the command line using `curl`:\n",
    "        * Type on one line `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_linux.yml > environment_linux.yml` and press Enter (or `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_mac.yml > environment_mac.yml` on Mac)\n",
    "    * Type in `conda env create -f environment_linux.yml` and press Enter (or `conda env create -f environment_mac.yml` on Mac)\n",
    "    * Anaconda will then run all the necessary command to download the various packages above using conda and pip\n",
    "    * Sometimes when you see the progress bar for certain libraries it will appear to stall, but you can sometimes try pressing \"y\" to make it continue\n",
    "    * At the end of the process you'll have a Python 3.6 conda environment with the name 'py38class' which you can use, with all the packages you'll need for this course, such as pandas, numpy etc.\n",
    "    * Note that you may need to put in the full path for wherever you downloaded the `environment_linux.yml` (or `environment_mac.yml`) file or you can simply `cd` to that folder and then run the above conda command\n",
    "    * Typically on Mac OS X when you save down files from your internet browser they will be in the user's home downloads folder, which you can get to by typing\n",
    "        `cd ~/Downloads` and then pressing Enter\n",
    "    \n",
    "* The `environment_linux.yml` file (or `environment_mac.yml` or similar name) basically has all the instructions required to recreate a conda environment\n",
    "* To create your own `environment.yml` file (for backup purposes, or if you'd like to distribute your conda environment) run the below command in your Anaconda Prompt\n",
    "    * `conda env export > environment_linux.yml` on Linux\n",
    "    * `conda env export > environment_mac.yml` on Mac"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:12:06.412884Z",
     "start_time": "2019-10-01T16:12:06.404503Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Linux/Mac: Installing a conda environment for Linux/Mac (slower method)\n",
    "\n",
    "A conda environment is a separate version of Python, where you can install all your own Python environments. For Linux \n",
    "and Mac, we'll install more libraries, which you might need to use later (some of which aren't fully supported by Windows). Note that underneath `create_conda_env_linux.sh` (or Mac) are using [mamba](https://github.com/mamba-org/mamba) - which is a faster implementation of conda to install some libraries versus conda. If you have any difficulties with mamba, you might need to change references in `create_conda_env_linux.sh` (or Mac) where mamba is installing libraries to conda. If this method still doesn't work/or hangs, skip to the section \"Installing a conda environment if all else fails\".\n",
    "\n",
    "### Download `create_conda_env_mac.sh` or `create_conda_env_linux.sh`\n",
    "\n",
    "* Open a Terminal window (usually a black window icon on both Linux and Mac)\n",
    "  * (Method 1) Download `create_conda_env_mac.sh` or `create_conda_env_linux.sh`\n",
    "    * Open `create_conda_env_mac.sh` from GitHub by clicking this [link](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_mac.sh) or `create_conda_env_linux.sh` from GitHub by clicking this [link](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_linux.sh) or \n",
    "    * Copy all the text from your browser\n",
    "    * Open up a text editor\n",
    "    * Paste the text\n",
    "    * Create a folder `/Users/<your-username>/pythoncourse` for Mac or `/home/<your-username>/pythoncourse` for Linux\n",
    "    * Save text file as `create_conda_env_linux.bat` in the `/Users/<your-username>/pythoncourse` for Mac or `/home/<your-username>/pythoncourse` for Linux\n",
    "  * (Method 2) Alternatively, you can download `create_conda_env_mac.sh` or `create_conda_env_linux.sh` from the Terminal window\n",
    "    * Create a folder `/Users/<your-username>/pythoncourse` for Mac or `/home/<your-username>/pythoncourse` for Linux\n",
    "    * Type the below command (typed all on one line in a Terminal window then press Enter):\n",
    "      * `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_mac.sh > create_conda_env_mac.sh`   \n",
    "      * `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_linux.sh > create_conda_env_linux.sh`   \n",
    "\n",
    "### Kick off installation\n",
    "\n",
    "#### Make sure Anaconda is on your Mac or Linux path\n",
    "\n",
    "* In a Terminal window (usually a black window icon on both Linux and Mac)\n",
    "    * Note, conda might already be on your path - you can check this by typing \n",
    "        `conda activate` and pressing enter, \n",
    "    * Check if you get an error or not. On Windows the installer will have a default option to add Anaconda to your path. If you don't get an error, then conda is already on your path, and you can skip to the section \"Run Anaconda environment installation\"\n",
    "  * Type in (or wherever you installed Anaconda sometimes on Mac it can be at `/Users/<your-username>/opt/anaconda3/bin`) and press enter to run\n",
    "    * `cd /Users/<your-username>/anaconda3/bin` for Mac\n",
    "    * `cd /home/<your-username>/anaconda3/bin` for Linux\n",
    "  * conda is in this folder\n",
    "* On Linux/Mac, it is recommend to add the Anaconda folder to your path\n",
    "* You can temporarily add paths in Linux/Mac using the following command in the terminal (change the path to wherever you installed Anaconda)\n",
    "    * For Mac: `export PATH=/Users/<your-username>/anaconda3/bin:$PATH`\n",
    "    * For Linux: `export PATH=/home/<your-username>/anaconda3/bin:$PATH`\n",
    "* To permanently add folders to your path by editing a file like `.bashrc` which is in your home folder\n",
    "    * http:https://osxdaily.com/2015/07/28/set-enviornment-variables-mac-os-x/ for Mac\n",
    "    * https://opensource.com/article/17/6/set-path-linux for Linux\n",
    "    \n",
    "#### Run Anaconda environment installation\n",
    "\n",
    "* Make the scripts executable\n",
    "    * For Mac: `chmod +x /Users/<your-username>/pythoncourse/create_conda_env_mac.sh`\n",
    "    * For Linux: `chmod +x /home/<your-username>/pythoncourse/create_conda_env_linux.sh`\n",
    "* Then run\n",
    "    * For Mac: `./Users/<your-username>/pythoncourse/create_conda_env_mac.sh`\n",
    "    * For Linux: `./home/<your-username>/pythoncourse/create_conda_env_linux.sh`\n",
    "* It will likely take a long time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Windows/Linux/Mac: Installing a conda environment if all else fails\n",
    "\n",
    "If either the faster or slower methods for installing a conda environment fail, you can try to create something similar to the py38class environment in a more manual way. Then later, you will likely to need to install libraries as you need them through the course.\n",
    "\n",
    "In your Anaconda Prompt on Windows or Terminal on Linux/Mac, run the following commands.\n",
    "\n",
    "* Switch to the `base` environment\n",
    "  * `conda activate`\n",
    "  \n",
    "* Remove any existing `py38class` environment\n",
    "  * `conda remove -n py38class --all --yes`\n",
    "  \n",
    "* Create a barebones `py38class` environment (if this hangs, we'll just have to use our `base` environment\n",
    "  * `conda create -n py38class python=3.8`\n",
    "  \n",
    "* Activate the new `py38class` environment (if it managed to install, otherwise we'll just install everything in our `base` environment, which isn't ideal, but will at least get us the libraries we need), install the `anaconda` libraries, which include things like Pandas, NumPy etc. and install the minimal set of Cuemacro finance libraries to get you started, and do pip install as needed with additional libraries\n",
    "  * `conda activate py38class`\n",
    "  * `conda install anaconda` (and if you find it hangs remove this line)\n",
    "  * `pip install pandas==1.2.3 finmarketpy chartpy findatapy cufflinks kaleido dash plotly`\n",
    "  * On Mac or Windows `pip install xlwings`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problems with getting correct Python to run in Jupyter notebook\n",
    "\n",
    "When running a Jupyter notebook if you get a kernal error displaed and in the command line it shows an error like `Kernal Error - FileNotFoundError: [WinError 2] The system cannot find the file specified`. Basically, this means that Jupyter can't find the correct Python under your conda environment (eg. `py38class`).\n",
    "\n",
    "One fix is to search for `kernel.json` under your Anaconda folder, and check it has the correct path for the Python under your conda environment. Eg. for `py38class` we would likely find it in `<<Whatever your Anaconda Folder Is>>/py38class/share/jupyter/kernels/python3/kernel.json` and below is a sample file.\n",
    "\n",
    "`{\n",
    " \"argv\": [\n",
    "  \"c:/Anaconda3/envs/py38class\\\\python.exe\",\n",
    "  \"-m\",\n",
    "  \"ipykernel_launcher\",\n",
    "  \"-f\",\n",
    "  \"{connection_file}\"\n",
    " ],\n",
    " \"display_name\": \"Python 3 (ipykernel)\",\n",
    " \"language\": \"python\",\n",
    " \"metadata\": {\n",
    "  \"debugger\": true\n",
    " }\n",
    "}`\n",
    "\n",
    "Once, you've made these changes you'll need to restart Jupyter.\n",
    "\n",
    "Alternatively, you might need to install the `ipykernel` from the command line. To do this in the Anaconda Prompt type in the following:\n",
    "\n",
    "* `conda activate py38class`\n",
    "* `python -m ipykernel install --user --name=py38class`\n",
    "\n",
    "More details can be found on https://github.com/jupyter/notebook/issues/4079 and https://towardsdatascience.com/get-your-conda-environment-to-show-in-jupyter-notebooks-the-easy-way-17010b76e874"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:04:01.263522Z",
     "start_time": "2019-10-01T16:04:01.258640Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Download and install PyCharm\n",
    "\n",
    "* If you running Python locally, you can try to install PyCharm, which makes it easier to develop Python code\n",
    "* You can download and install PyCharm Community from https://www.jetbrains.com/pycharm/download\n",
    "* There are versions for PyCharm for Windows, Linux and Mac operating systems.\n",
    "* PyCharm IDE makes it easier to write and run Python code\n",
    "* In File / Settings - you will likely need to set PyCharm to use your py38class environment, which is likely at the following locations\n",
    "    * Windows: `C:\\Users\\<your-username>\\Anaconda3\\envs\\py38class`\n",
    "    * macOS: `/Users/<your-username>/anaconda3/envs/py38class`\n",
    "    * Linux: `/home/<your-username>/anaconda3/envs/py38class`\n",
    "* PyCharm will create a workspace for you, where you can place your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problems with `conda activate`\n",
    "\n",
    "Sometimes you might find that running `conda activate` in the shell (eg. terminal) doesn't seem to return you to the `base` environment or that running `conda activate py38class` doesn't change your environment.\n",
    "\n",
    "To fix this you can try running `conda init` in your shell, and then close it and open it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Google Colab with Jupyter\n",
    "\n",
    "If you do not want to install Anaconda on your own machine (or if you don't have the correct permissions to do so), you can instead use Google Colab, which gives you a Jupyter notebook in the cloud. This can be a good solution for those wanting to learn Python. You'll need a Google account, otherwise you can't save down your notebooks. You can access it at https://colab.research.google.com/\n",
    "\n",
    "If you find there are libraries which aren't available, you can install these in Google Colab (as with any Jupyter notebook) using `!pip` inside the notebook. Below, we have a number of useful libraries to get you started. You might need to run more than these. Note that you might need to re-run pip on Google Colab every so often, because the server gets restarted, and the library installations will be lost. \n",
    "\n",
    "There are several other alternatives to Google Colab, such as https://cocalc.com/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \\\n",
    "  redis pathos pyarrow==4.0.0 pandas==1.2.3 quandl \\\n",
    "  finmarketpy chartpy findatapy \\\n",
    "  cufflinks==0.17.3 kaleido \\\n",
    "  plotly==4.14.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional installations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Download and install Git (optional)\n",
    "\n",
    "Git is version control software, which maybe useful to install some Python libraries we'll use (in practice you can install these without Git, but they might not be the latest versions). It's also worth understanding how to use version control if you want to code later! You can download and install Git for Windows, Linux and Mac operating systems from https://git-scm.com/downloads\n",
    "\n",
    "Note that for Linux, you can install from the command line but the syntax depends on your Linux distribution https://git-scm.com/download/linux discusses this in some detail.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "### GPU versions of TensorFlow and PyTorch (optional)\n",
    "\n",
    "If you want to use your GPU for certain operations in particular for PyTorch and TensorFlow, you may need to update your NVIDIA graphics driver. First check if you have a GPU accelerated graphics card which supports CUDA (usually most newer NVIDIA graphics cards). This means you can install GPU accelerated versions of machine libraries like TensorFlow and PyTorch. \n",
    "\n",
    "To do this, you need to manually install various CUDA libraries. For full details on how to install these see https://www.tensorflow.org/install/gpu (both for Windows and Linux). Once you've done that you can edit the scripts below where indicated, so the GPU enabled Python versions of PyTorch and TensorFlow are installed (rather than the CPU version). Note, the CPU versions work fine, but will be slower and by default the environments below install CPU versions for maximum compatibility in case you don't have an NVIDIA card. If you are running on the cloud, you need to check that the cloud machine you are using has a GPU. Typically free instances are CPU only."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Bloomberg to access data via API (optional)\n",
    "\n",
    "It is helpful to install Bloomberg's blpapi Python library if you can (and have a Bloomberg account and have a Bloomberg Terminal with Windows). \n",
    "\n",
    "The `py38class` environment for Windows already contains blpapi, so you don't need to do this step!\n",
    "\n",
    "However, if you want to install it without conda, you follow the instructions at https://github.com/cuemacro/findatapy/blob/master/BLOOMBERG.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing tabula-py and pytesseract (optional)\n",
    "\n",
    "If you'd like to use tabula-py (extracting tables from PDF) and pytesseract (for doing optical character recognition), as well as install the Python libraries with pip, you'll also need to some further steps:\n",
    "\n",
    "tabula-py uses the Java runtime underneath, hence it needs the Java runtime installed on your path.\n",
    "\n",
    "* Tabula installation on Windows - https://tabula-py.readthedocs.io/en/latest/getting_started.html#get-tabula-py-working-windows-10\n",
    "\n",
    "pytesseract is a wrapper for Tesseract, which needs to be installed first\n",
    "\n",
    "* Tesseract installation on Windows - https://github.com/UB-Mannheim/tesseract/wiki. \n",
    "* Tesseract installation on Linux at https://tesseract-ocr.github.io/tessdoc/Home.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing additional libraries for data storage and AWS (optional)\n",
    "\n",
    "By default Arctic is installed by `py38class` which makes it easy to install Pandas DataFrames. However, if you want to use it with Pandas 1.2.3, you might need to install the latest version from the GitHub repo for Arctic (https://github.com/man-group/arctic), by running the following commands:\n",
    "\n",
    "* `conda activate py38class`\n",
    "* `pip uninstall arctic`\n",
    "* `pip install git+https://github.com/man-group/arctic.git` (Note this requires Git to be installed as discussed above)\n",
    "\n",
    "Also to use S3 to store Pandas DataFrames using findatapy, you'll need to install `s3fs`\n",
    "\n",
    "* `conda activate py38class`\n",
    "* `pip install s3fs`\n",
    "\n",
    "More broadly to setup using AWS services via Python, you'll need to install the AWS CLI and set your credentials. This [Jupyter notebook](https://github.com/cuemacro/finmarketpy/blob/master/finmarketpy_examples/finmarketpy_notebooks/s3_bucket_example.ipynb) examples how to setup AWS CLI and also how to use S3 with findatapy to store DataFrames as Parquet files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test your Python installation by running this.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can test your Python installation, by starting the Anaconda prompt (switching to the right conda environment) and then starting Jupyter (in Google Colab, you'll start a Jupyter notebook via it's own interface) using the below commands. \n",
    "\n",
    "Note, that you will likely have to change the default notebook-dir parameter (or you can just omit it, in which case Jupyter will use the current working directory)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    conda activate py38class\n",
    "    jupyter notebook --notebook-dir='e:/cuemacro/pythoncourse/pythoncourse/notebooks'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then try running the below Python code in a Jupyter notebook or the Python interpreter to see if some of the libraries we've installed work. This is not an exhaustive test, but only a few which we'll use a lot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-22T18:20:06.015735Z",
     "start_time": "2021-04-22T18:20:00.933053Z"
    }
   },
   "outputs": [],
   "source": [
    "import chartpy\n",
    "import quandl\n",
    "import finmarketpy\n",
    "import findatapy\n",
    "import pandas\n",
    "import numpy\n",
    "import plotly"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "500.75px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}