pythoncourse/installation/installing_anaconda_and_pycharm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# Installing Anaconda and PyCharm\n",
    "\n",
    "Saeed Amen - Copyright Cuemacro 2020 - https://www.cuemacro.com - saeed@cuemacro.com\n",
    "\n",
    "In this notebook, we discuss how to install the Anaconda distribution of Python and other applications you need to run do development in Python, which will be useful for doing financial analysis in Python with libraries including the Cuemacro libraries, finmarketpy, findatapy and chartpy. \n",
    "\n",
    "You can also download a lot of this material from https://github.com/cuemacro/teaching - in particular the scripts for installing the conda environments. We'll also be installing a lot of other useful libraries for machine learning, natural language processing etc. \n",
    "\n",
    "We'll also make other suggestions so you can do financial data analysis, such as making sure that your firewalls allow access to sites like Quandl."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make sure firewalls are open to allow access to market data\n",
    "\n",
    "This is an important one! Make sure that you can download data via Python for libraries like Quandl and also to install Python. \n",
    "\n",
    "* Once you've done your Python installation (and have signed up for a Quandl account and API key for free), try running this code below to check. \n",
    "* Your firewall should also allow you to have access to allow the downloading of packages via conda and pip (if it isn't then you can't do much else, although in some circumstances you might be able to change the pip download URL to a site that does work)\n",
    "* If opening up firewalls is too much of an issue or you don't have admin rights to install Python locally, then one easier way to get started with Python for learning purposes, is to use something Azure Notebooks (see below). \n",
    "* This will however, at the very least need the website https://notebooks.azure.com/ to be accessible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quandl.ApiConfig.api_key = \"TYPE YOUR KEY HERE\"\n",
    "\n",
    "# Download data for WTI Crude oil from the EIA database on Quandl (US Department of Energy)\n",
    "# We have looked up this ticker earlier on Quandl's website\n",
    "df_wti = pd.DataFrame(quandl.get(\"EIA/PET_RWTC_D\", start_date=\"2010-01-01\", end_date=\"2019-12-31\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Download and install Anaconda\n",
    "\n",
    "You can download install the Anaconda distribution for Python from https://www.anaconda.com/distribution \n",
    "\n",
    "There are versions of Anaconda for Windows, Linux and Mac operating systems.\n",
    "\n",
    "By default the Anaconda will be installed at the following folder, which will depend on your username. Make sure you keep a note of where you installed Anaconda for later, in particular when you need to tell Anaconda where you installed the distribution!\n",
    "\n",
    "* Windows: `C:\\Users\\<your-username>\\Anaconda3\\`\n",
    "* macOS: `/Users/<your-username>/anaconda3`\n",
    "* Linux: `/home/<your-username>/anaconda3`\n",
    "\n",
    "Anaconda comes with many data science focused Python libraries. However, we'll still need to install quite a few other extra ones. Also in some instances, we'll install a different versions of certain libraries (including an earlier one for Pandas).\n",
    "\n",
    "On Windows It is also recommended to add some Anaconda folders (should look similar to the below) to your Windows PATH (the Anaconda installer usually has a setting you can tick for this, but if that doesn't work do it in your environment variables in Control Panel). If this isn't set you can have issues when running certain libraries like xlwings.\n",
    "\n",
    "* `C:\\Users\\<your-username>\\Anaconda3`\n",
    "* `C:\\Users\\<your-username>\\Anaconda3\\Scripts`\n",
    "* `C:\\Users\\<your-username>\\Anaconda3\\Library\\bin`\n",
    "\n",
    "You might need to allow Win32 long paths, otherwise Windows will restrict the number of folders on your path, can end up being very long with Python installations! See https://stackoverflow.com/questions/26155135/node-npm-windows-file-paths-are-too-long-to-install-packages/37528731#37528731 on how to set this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Download and install Git (optional)\n",
    "\n",
    "Git is version control software, which maybe useful to install some Python libraries we'll use (in practice you can install these without Git, but they might not be the latest versions). It's also worth understanding how to use version control if you want to code later! You can download and install Git for Windows, Linux and Mac operating systems from https://git-scm.com/downloads\n",
    "\n",
    "Note that for Linux, you can install from the command line but the syntax depends on your Linux distribution https://git-scm.com/download/linux discusses this in some detail.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "## GPU versions of TensorFlow and PyTorch (optional)\n",
    "\n",
    "If you want to use your GPU for certain operations in particular for PyTorch and TensorFlow, you may need to update your NVIDIA graphics driver. First check if you have a GPU accelerated graphics card which supports CUDA (usually most newer NVIDIA graphics cards). This means you can install GPU accelerated versions of machine libraries like TensorFlow and PyTorch. \n",
    "\n",
    "To do this, you need to manually install various CUDA libraries. For full details on how to install these see https://www.tensorflow.org/install/gpu (both for Windows and Linux). Once you've done that you can edit the scripts below where indicated, so the GPU enabled Python versions of PyTorch and TensorFlow are installed (rather than the CPU version). Note, the CPU versions work fine, but will be slower and by default the environments below install CPU versions for maximum compatibility in case you don't have an NVIDIA card. If you are running on the cloud, you need to check that the cloud machine you are using has a GPU. Typically free instances are CPU only."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Bloomberg to access data via API (optional)\n",
    "\n",
    "It is helpful to install Bloomberg's blpapi Python library if you can (and have a Bloomberg account and have a Bloomberg Terminal with Windows). It is recommended to install in your conda environment if it running locally. Full details on how to install blpapi can be found at https://github.com/cuemacro/findatapy/blob/master/BLOOMBERG.md (without using conda). The Windows script already includes the conda installation of blpapi, which makes it a lot quicker!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing tabula-py and pytesseract\n",
    "\n",
    "If you'd like to use tabula-py (extracting tables from PDF) and pytesseract (for doing optical character recognition), as well as install the Python libraries with pip, you'll also need to some further steps:\n",
    "\n",
    "tabula-py uses the Java runtime underneath, hence it needs the Java runtime installed on your path.\n",
    "\n",
    "* Tabula installation on Windows - https://tabula-py.readthedocs.io/en/latest/getting_started.html#get-tabula-py-working-windows-10\n",
    "\n",
    "pytesseract is a wrapper for Tesseract, which needs to be installed first\n",
    "\n",
    "* Tesseract installation on Windows - https://github.com/UB-Mannheim/tesseract/wiki. \n",
    "* Tesseract installation on Linux at https://tesseract-ocr.github.io/tessdoc/Home.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing a conda environment for Windows from YML file (quickest method - recommended)\n",
    "\n",
    "We can download [environment_windows.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_windows.yml) from our browser or via `curl` in the command line. We can use that YML file to create our conda environment with all the necessary packages. This also installs the exact same libraries versions that I have (and reduces the likelihood of version conflicts between libraries).\n",
    "\n",
    "* Open up the Anaconda Prompt (should be in the Start Menu)\n",
    "* In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in `conda activate` and press Enter to exit the current conda environment, then `conda remove -n py37class --all --yes` and press Enter to remove any existing environments called `py37class`\n",
    "    * If you haven't already downloaded the `environment_windows.yml` file you can do it from the command line using `curl`:\n",
    "        * Type `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_windows.yml > environment_windows.yml` and press Enter\n",
    "    * Type in `conda env create -f environment_windows.yml` and press Enter\n",
    "    * Anaconda will then run all the necessary command to download the various packages above using conda and pip\n",
    "    * At the end of the process you'll have a Python 3.7 conda environment with the name 'py37class' which you can use, with all the packages you'll need for this course, such as pandas, numpy etc.\n",
    "    * Note that you may need to put in the full path for wherever you downloaded the `environment_windows.yml` file or you can simply `cd` to that folder and then run the above conda command\n",
    "    * Typically when you download files from your internet browser, Windows will usually save it in the users downloads folder, this is usually accessable by doing\n",
    "        `cd \\Users\\<your-username>\\Downloads` and then pressing Enter\n",
    "    \n",
    "* The `environment_windows.yml` file (or similar name) basically has all the instructions required to recreate a conda environment\n",
    "* To create your own `environment.yml` file (for backup purposes, or if you'd like to distribute your conda environment) run the below command in your Anaconda Prompt\n",
    "    * `conda env export > environment_windows.yml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:00:00.009779Z",
     "start_time": "2019-10-01T16:00:00.000993Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Installing a conda environment for Windows (slower method)\n",
    "\n",
    "A conda environment is a separate version of Python, where you can install all your own Python environments. This is a slower way to do it, but this makes it easier to change the versions of the libraries (note, might not be 100% same versions as my libraries!)\n",
    "\n",
    "* Download \n",
    "* Open up Notepad\n",
    "* Copy the below text\n",
    "* Create a folder `C:\\pythoncourse` (or if you create it elsewhere, keep a note of that)\n",
    "* Save file `create_conda_env_windows.bat` (see download link below) in the `C:\\pythoncourse` folder\n",
    "* Open up the Anaconda Prompt (should be in the Start Menu)\n",
    "* In this prompt, your Anaconda folder will on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "* Type in\n",
    "    `cd\\`\n",
    "* This will change directory to the `C:\\` drive\n",
    "* Then run\n",
    "    `cd\\pythoncourse`\n",
    "* Then run\n",
    "    `create_conda_env_windows.bat`\n",
    "* It will likely take a long time, and you might need to periodically click yes in Windows to allower the installer to change your settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### [Download create_conda_env_windows.bat from GitHub](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_windows.bat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing xlwings addin in Excel on Windows (or Mac)\n",
    "\n",
    "Both of the above should install the xlwings Python library. You also need to add the xlwings addin to Excel too. Instructions for this can be found at https://docs.xlwings.org/en/stable/addin.html (xlwings is also supported on Excel for Mac, although the functionality may differ). It won't work on Linux, given there's no Linux version of Excel). Usually this will involve, closing Excel, then running the following commands in your Anaconda Prompt, before restarting Excel:\n",
    "\n",
    "    conda activate py37class\n",
    "    xlwings addin install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing a conda environment for Linux or Mac from YAML file (quickest method - recommended)\n",
    "\n",
    "We can download [environment_linux.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_linux.yml) if we're installing on Linux or [environment_mac.yml (click to download)](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_mac.yml) for Mac from our browser or via `curl` in the command line. We can use that YML file to create our conda environment with all the necessary packages . This should be a lot faster to run. This also installs the exact same libraries versions that I have (and reduces the likelihood of version conflicts between libraries).\n",
    "\n",
    "* Open up a terminal window (making sure that Anaconda is on your path) on Linux or Mac\n",
    "* In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in `conda activate` to exit the current conda environment and press Enter, then `conda remove -n py37class --all --yes` and press Enter to remove any existing environments called `py37class`\n",
    "    * If you haven't already downloaded the `environment_linux.yml` or `environment_mac.yml` file you can do it from the command line using `curl`:\n",
    "        * Type `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_linux.yml > environment_linux.yml` and press Enter (or `curl https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_mac.yml > environment_mac.yml` on Mac)\n",
    "    * Type in `conda env create -f environment_linux.yml` and press Enter (or `conda env create -f environment_mac.yml` on Mac)\n",
    "    * Anaconda will then run all the necessary command to download the various packages above using conda and pip\n",
    "    * At the end of the process you'll have a Python 3.6 conda environment with the name 'py37class' which you can use, with all the packages you'll need for this course, such as pandas, numpy etc.\n",
    "    * Note that you may need to put in the full path for wherever you downloaded the `environment_linux.yml` (or `environment_mac.yml`) file or you can simply `cd` to that folder and then run the above conda command\n",
    "    * Typically on Mac OS X when you save down files from your internet browser they will be in the user's home downloads folder, which you can get to by typing\n",
    "        `cd ~/Downloads` and then pressing Enter\n",
    "    \n",
    "* The `environment_linux.yml` file (or `environment_mac.yml` or similar name) basically has all the instructions required to recreate a conda environment\n",
    "* To create your own `environment.yml` file (for backup purposes, or if you'd like to distribute your conda environment) run the below command in your Anaconda Prompt\n",
    "    * `conda env export > environment_linux.yml` on Linux\n",
    "    * `conda env export > environment_mac.yml` on Mac"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:12:06.412884Z",
     "start_time": "2019-10-01T16:12:06.404503Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Installing a conda environment for Linux and Mac (slower method)\n",
    "\n",
    "A conda environment is a separate version of Python, where you can install all your own Python environments. For Linux \n",
    "and Mac, we'll install more libraries, which you might need to use later (some of which aren't fully supported by Windows).\n",
    "\n",
    "* Download \n",
    "* Open up a text editor\n",
    "* Copy the below text\n",
    "* Create a folder `/Users/<your-username>/pythoncourse` for Mac or `/home/<your-username>/pythoncourse` for Linux\n",
    "* Save file `create_conda_env_linux.sh` or `create_conda_env_mac.sh` (see download link below) in the pythoncourse folder\n",
    "* Open a Terminal window (usually a black window icon on both Linux and Mac)\n",
    "* Type in (or wherever you installed Anaconda) and press enter to run\n",
    "    * `cd /Users/<your-username>/anaconda3/bin` for Mac\n",
    "    * `cd /home/<your-username>/anaconda3/bin` for Linux\n",
    "* conda is in this folder\n",
    "* Note, conda might already be on your path - you can check this by typing `conda` and pressing enter, to see if you get an error. On Windows\n",
    "the installer will have a default option to add Anaconda to your path\n",
    "* On Linux/Mac, it is recommend to add the Anaconda folder to your path\n",
    "* You can temporarily add paths in Linux/Mac using the following command in the terminal (change the path to wherever you installed\n",
    "Anaconda)\n",
    "    * `export PATH=/Users/<your-username>/anaconda3/bin:$PATH` for Mac\n",
    "    * `export PATH=/home/<your-username>/anaconda3/bin:$PATH` for Linux\n",
    "* To permanently add folders to your path by editing a file like `.bashrc` which is in your home folder\n",
    "    * http:https://osxdaily.com/2015/07/28/set-enviornment-variables-mac-os-x/ for Mac\n",
    "    * https://opensource.com/article/17/6/set-path-linux for Linux\n",
    "* Then run\n",
    "    * `./Users/<your-username>/pythoncourse/create_conda_env_mac.sh` for Mac\n",
    "    * `./home/<your-username>/pythoncourse/create_conda_env_linux.sh` for Linux\n",
    "* It will likely take a long time, and you might need to periodically click yes also Windows to change settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:13:50.842110Z",
     "start_time": "2019-10-01T16:13:50.828861Z"
    },
    "pycharm": {}
   },
   "source": [
    "### [Download create_conda_env_linux.sh from GitHub for Linux](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_linux.sh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Download create_conda_env_mac.sh from GitHub for Mac](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/create_conda_env_mac.sh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:04:01.263522Z",
     "start_time": "2019-10-01T16:04:01.258640Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Download and install PyCharm\n",
    "\n",
    "* If you running Python locally, you can try to install PyCharm, which makes it easier to develop Python code\n",
    "* You can download and install PyCharm Community from https://www.jetbrains.com/pycharm/download\n",
    "* There are versions for PyCharm for Windows, Linux and Mac operating systems.\n",
    "* PyCharm IDE makes it easier to write and run Python code\n",
    "* In File / Settings - you will likely need to set PyCharm to use your py37class environment, which is likely at the following locations\n",
    "    * Windows: `C:\\Users\\<your-username>\\Anaconda3\\envs\\py37class`\n",
    "    * macOS: `/Users/<your-username>/anaconda3/envs/py37class`\n",
    "    * Linux: `/home/<your-username>/anaconda3/envs/py37class`\n",
    "* PyCharm will create a workspace for you, where you can place your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Google Colab with Jupyter\n",
    "\n",
    "If you do not want to install Anaconda on your own machine (or if you don't have the correct permissions to do so), you can instead use Google Colab, which gives you a Jupyter notebook in the cloud. This can be a good solution for those wanting to learn Python. You'll need a Google account, otherwise you can't save down your notebooks. You can access it at https://colab.research.google.com/\n",
    "\n",
    "If you find there are libraries which aren't available, you can install these in Google Colab (as with any Jupyter notebook) using `!pip` inside the notebook. Below, we have a number of useful libraries to get you started. Note that you might need to re-run pip on Google Colab every so often, because the server gets restarted, and the library installations will be lost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \\\n",
    "  redis pathos pyarrow==1.0.1 pandas==1.0.5 quandl \\\n",
    "  finmarketpy chartpy findatapy \\\n",
    "  cufflinks==0.17.3 kaleido \\\n",
    "  plotly==4.9.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Microsoft Azure Notebooks with Jupyter\n",
    "\n",
    "If you do not want to install Anaconda on your own machine (or if you don't have the correct permissions to do so), you can instead use Microsoft Azure Notebooks, which gives you a Jupyter notebook in the cloud. This can be a good solution for those wanting to learn Python. Note that Microsoft are phasing this out so I'd recommend Google Colab notebooks at present.\n",
    "\n",
    "### Creating your first Jupyter notebook in Microsoft Azure Notebooks\n",
    "\n",
    "Below we outline the steps to creating your first Jupyter notebook in Microsoft Azure Notebooks.\n",
    "\n",
    "* Go to https://notebooks.azure.com/ and sign in with your Microsoft account\n",
    "    * If you do not already have a Microsoft account eg. for Outlook.com you can sign up for a free account or on the Azure notebook website\n",
    "* At https://notebooks.azure.com/TYPEYOURUSERNAME/projects you will be able to see the projects you currently have\n",
    "* Create a new project eg. named `pythoncourse` in which to place all your Jupyter notebooks\n",
    "\n",
    "### Creating your conda environment in Microsoft Azure Notebooks (harder)\n",
    "\n",
    "* You'll also need to set this new project to use the conda environment ready for the course, so it has all the right Python libraries ready for you to use\n",
    "    * At https://docs.microsoft.com/en-us/azure/notebooks/quickstart-create-jupyter-notebook-project-environment you can find additional details on how to create a conda environment for Azure Notebooks (using an environment.yml file)\n",
    "    * You'll need the [environment_azure_linux.yml](https://raw.githubusercontent.com/cuemacro/teaching/master/pythoncourse/installation/environment_azure_linux.yml) and then upload it to your newly made project\n",
    "    * Note, some libraries may not work in Azure notebooks such as the Bloomberg Python API\n",
    "    * GPU versions of some of the libraries may not work, unless you set to Azure notebooks to use a (paid-for) GPU enabled workstation\n",
    "* There are other similar solutions for Jupyter in the cloud, eg. Google Colab etc. which you might also consider\n",
    "\n",
    "### Alternatively in your Azure Jupyter notebook to setup your Python environment (easier)\n",
    "\n",
    "* If you're having trouble getting Azure to setup your conda environment with the above method (often it isn't clear when Azure has setup the conda environment from above). In that case, it is suggested to do the below, which will also execute a lot quicker\n",
    "* Here we are trying to only install some of the most important additional packages, as Azure already comes with many to begin, and this will reduce the download/install time\n",
    "* We've also excluded some which are likely to are less stable in the Azure environment (or require a lot of other dependencies) \n",
    "* We've rewritten to use pip, as conda can take a while on slower machines (except for packages which are difficult to install with pip like plotly-orca, which is optional)\n",
    "* Then run the below commands in your Jupyter notebook (this will likely take a long time, usually a couple of minutes)\n",
    "* Make sure you are running Azure notebooks in Python 3.6 (not 3.5)\n",
    "* Note, that you might to run these every so often, because after 60 minutes of inactivity Azure will kill your notebook instance if you are using the free account on Azure Notebooks (and hence any Python library installations you do will also disappear)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A selection of libraries (note: this isn't everything, you may need to \"pip\" install extra libraries)\n",
    "!pip install \\\n",
    "  redis pathos pyarrow==1.0.1 pandas==1.0.5 wordcloud quandl \\\n",
    "  vaex-core vaex-viz vaex-jupyter vaex-server vaex-hdf5 vaex-astro vaex-distributed pyspark py4j \\\n",
    "  spacy celery==4.4.0 celery[redis] celery[msgpack] msgpack-python \\\n",
    "  fxcmpy alpha_vantage yfinance twython newspaper3k seasonal pdfminer.six vaderSentiment gensim textacy \\\n",
    "  transformers \\\n",
    "  finmarketpy chartpy findatapy \\\n",
    "  cufflinks==0.17.3  kaleido \\\n",
    "  plotly==4.9.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or rather than running the above commands, you can try creating a full conda environment in Azure from the yml file (although from experience this is less likely to work, and is much slower and it's difficult to activate later)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!conda env create -f environment_azure_linux.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check if your Azure Python package installation has worked by importing Pandas and checking the version is indeed 0.24.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-20T12:35:44.464099Z",
     "start_time": "2020-02-20T12:35:43.650407Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.24.2'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas\n",
    "pandas.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Direct compute on a paid Azure Jupyter instance (Data Science Virtual Machine)\n",
    "\n",
    "The free version of Azure notebooks has a number of limitations. In particular, that it resets every hour and only has 4GB RAM. You can instead create paid instances Data Science Virtual Machine for Linux (Ubuntu) on Azure (DSVM), which is a Linux box for a number of different CPUs/RAM which are much more powerful. You can configure the machine as want, and the configuration will be stored for as long as you have the instance. Do make sure to power down the machine, once you are finished using it. This won't wipe the hard disk, but it will mean you aren't paying for it when it's idle.\n",
    "\n",
    "Whilst Azure 3.6 ML from the DSVM already has many libraries installed, eg. TensorFlow etc. You will still need to install additional Python libraries, but this can be done once. Script `update_azure_conda_env.sh` will attempt to install these extra libraries such as NLTK etc. and also a version of Pandas which should be more compatible.\n",
    "\n",
    "You can then connect your Azure notebook to run on this server via \"Direct compute\", where you need to specify the IP of the machine. At https://docs.microsoft.com/en-us/azure/notebooks/use-data-science-virtual-machine it gives details on how to do \"Direct compute\" on your Azure Notebook. If you get a powerful machine you should be able to host multiple users. However, you should add usernames/passwords for every user if you do this, so each user runs in their own instance.\n",
    "\n",
    "You may need to run `conda install ipykernel` to make a new conda environment appear in the Jupyter notebook. Make sure that you select the right Python kernel from the Jupyter menu (Kernel / Change Kernel)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test your Python installation by running this.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can test your Python installation, by starting the Anaconda prompt (switching to the right conda environment) and then starting Jupyter (in Azure Notebook, you'll start a Jupyter notebook via it's own interface) using the below commands. \n",
    "\n",
    "Note, that you will likely have to change the default notebook-dir parameter (or you can just omit it, in which case Jupyter will use the current working directory)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    conda activate py37class\n",
    "    jupyter notebook --notebook-dir='e:/cuemacro/pythoncourse/pythoncourse/notebooks'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then try running the below Python code in a Jupyter notebook or the Python interpreter to see if some of the libraries we've installed work. This is not an exhaustive test, but only a few which we'll use a lot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-04T18:38:51.129577Z",
     "start_time": "2019-12-04T18:38:51.123720Z"
    }
   },
   "outputs": [],
   "source": [
    "import chartpy\n",
    "import quandl\n",
    "import finmarketpy\n",
    "import findatapy\n",
    "import pandas\n",
    "import numpy\n",
    "import plotly"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}