pythoncourse/installation/installing_anaconda_and_pycharm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# Installing Anaconda and PyCharm\n",
    "\n",
    "Saeed Amen - Copyright Cuemacro 2019 - https://www.cuemacro.com - saeed@cuemacro.com\n",
    "\n",
    "In this notebook, we discuss how to install the Anaconda distribution of Python and other applications you need to run do development in Python, which will be useful for doing financial analysis in Python with libraries like finmarketpy. You can also down a lot of this material from https://github.com/cuemacro/teaching - in particular the scripts for installing the conda environments. We'll also be installing a lot of other useful libraries for machine learning, natural language processing etc. We'll also make other suggestions so you can do financial data analysis, such as making sure that your firewalls allow access to sites like Quandl."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make sure firewalls are open to allow access to market data\n",
    "\n",
    "This is an important one! Make sure that you can download data via Python for libraries like Quandl and also to install Python. \n",
    "\n",
    "* Once you've done your Python installation (and have signed up for a Quandl account and API key for free), try running this code below to check. \n",
    "* Your firewall should also allow you to have access to allow the downloading of packages via conda and pip (if it isn't then you can't do much else, although in some circumstances you might be able to change the pip download URL to a site that does work)\n",
    "* If opening up firewalls is too much of an issue or you don't have admin rights to install Python locally, then one easier way to get started with Python for learning purposes, is to use something Azure Notebooks (see below). \n",
    "* This will however, at the very least need the website https://notebooks.azure.com/ to be accessible)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quandl.ApiConfig.api_key = \"TYPE YOUR KEY HERE\"\n",
    "\n",
    "# Download data for WTI Crude oil from the EIA database on Quandl (US Department of Energy)\n",
    "# We have looked up this ticker earlier on Quandl's website\n",
    "df_wti = pd.DataFrame(quandl.get(\"EIA/PET_RWTC_D\", start_date=\"2010-01-01\", end_date=\"2019-12-31\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Download and install Anaconda\n",
    "\n",
    "You can download install the Anaconda distribution for Python from https://www.anaconda.com/distribution \n",
    "\n",
    "There are versions of Anaconda for Windows, Linux and Mac operating systems.\n",
    "\n",
    "By default the Anaconda will be installed at the following folder, which will depend on your username. Make sure you keep a note of where you installed Anaconda for later, in particular when you need to tell Anaconda where you installed the distribution!\n",
    "\n",
    "* Windows: `C:\\Users\\<your-username>\\Anaconda3\\`\n",
    "* macOS: `/Users/<your-username>/anaconda3`\n",
    "* Linux: `/home/<your-username>/anaconda3`\n",
    "\n",
    "Anaconda comes with many data science focused Python libraries. However, we'll still need to install quite a few other extra ones. Also in some instances, we'll install a different versions of certain libraries (including an earlier one for Pandas).\n",
    "\n",
    "On Windows It is also recommended to add some Anaconda folders (should look similar to the below) to your Windows PATH (the Anaconda installer usually has a setting you can tick for this, but if that doesn't work do it in your environment variables in Control Panel). If this isn't set you can have issues when running certain libraries like xlwings.\n",
    "\n",
    "* `C:\\Users\\<your-username>\\Anaconda3`\n",
    "* `C:\\Users\\<your-username>\\Anaconda3\\Scripts`\n",
    "* `C:\\Users\\<your-username>\\Anaconda3\\Library\\bin`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Download and install Git (optional)\n",
    "\n",
    "Git is version control software, which maybe useful to install some Python libraries we'll use (in practice you can install these without Git, but they might not be the latest versions). It's also worth understanding how to use version control if you want to code later! You can download and install Git for Windows, Linux and Mac operating systems from https://git-scm.com/downloads\n",
    "\n",
    "Note that for Linux, you can install from the command line but the syntax depends on your Linux distribution https://git-scm.com/download/linux discusses this in some detail.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "## GPU versions of TensorFlow and PyTorch (optional)\n",
    "\n",
    "If you want to use your GPU for certain operations in particular for PyTorch and TensorFlow, you may need to update your NVIDIA graphics driver. First check if you have a GPU accelerated graphics card which supports CUDA (usually most newer NVIDIA graphics cards). This means you can install GPU accelerated versions of machine libraries like TensorFlow and PyTorch. \n",
    "\n",
    "To do this, you need to manually install various CUDA libraries. For full details on how to install these see https://www.tensorflow.org/install/gpu (both for Windows and Linux). Once you've done that you can edit the scripts below where indicated, so the GPU enabled Python versions of PyTorch and TensorFlow are installed (rather than the CPU version). Note, the CPU versions work fine, but will be slower! If you are running on the cloud, you need to check that the cloud machine you are using has a GPU. Typically free instances are CPU only."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Bloomberg to access data via API (optional)\n",
    "\n",
    "It is helpful to install Bloomberg's blpapi Python library if you can (and have a Bloomberg account and have a Bloomberg Terminal with Windows). It is recommended to install in your conda environment if it running locally. Full details on how to install blpapi can be found at https://github.com/cuemacro/findatapy/blob/master/BLOOMBERG.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing tabula-py and pytesseract\n",
    "\n",
    "If you'd like to use tabula-py (extracting tables from PDF) and pytesseract (for doing optical character recognition), as well as install the Python libraries with pip, you'll also need to some further steps:\n",
    "\n",
    "tabula-py uses the Java runtime underneath, hence it needs the Java runtime installed on your path.\n",
    "\n",
    "* Tabula installation on Windows - https://tabula-py.readthedocs.io/en/latest/getting_started.html#get-tabula-py-working-windows-10\n",
    "\n",
    "pytesseract is a wrapper for Tesseract, which needs to be installed first\n",
    "\n",
    "* Tesseract installation on Windows - https://github.com/UB-Mannheim/tesseract/wiki. \n",
    "* Tesseract installation on Linux at https://tesseract-ocr.github.io/tessdoc/Home.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:00:00.009779Z",
     "start_time": "2019-10-01T16:00:00.000993Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Installing a conda environment for Windows (method 1)\n",
    "\n",
    "A conda environment is a separate version of Python, where you can install all your own Python environments.\n",
    "\n",
    "* Download \n",
    "* Open up Notepad\n",
    "* Copy the below text\n",
    "* Create a folder `C:\\pythoncourse` (or if you create it elsewhere, keep a note of that)\n",
    "* Save the file as `create_conda_env.bat` in the `C:\\pythoncourse` folder\n",
    "* Open up the Anaconda Prompt (should be in the Start Menu)\n",
    "* In this prompt, your Anaconda folder will on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "* Type in\n",
    "    `cd\\`\n",
    "* This will change directory to the `C:\\` drive\n",
    "* Then run\n",
    "    `cd\\pythoncourse`\n",
    "* Then run\n",
    "    `create_conda_env.bat`\n",
    "* It will likely take a long time, and you might need to periodically click yes in Windows to allower the installer to change your settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Copy the below to create_conda_env.bat for Windows (or [download it](create_conda_env.bat))\n",
    "\n",
    "```\n",
    "REM assumes that Anaconda has already been downloaded\n",
    "\n",
    "REM update conda\n",
    "call conda update -n base -c defaults conda --yes\n",
    "\n",
    "call conda activate\n",
    "\n",
    "REM remove any existing environment called py36class, and create a py36class with anaconda packages\n",
    "call conda remove -n py36class --all --yes\n",
    "call conda create -n py36class python=3.6 anaconda\n",
    "call conda activate py36class\n",
    "\n",
    "REM install from conda-forge\n",
    "call conda install -c conda-forge ^\n",
    "jupyterlab jupyter_contrib_nbextensions jupyter_nbextensions_configurator ipywidgets=7.5 ^\n",
    "redis-py=3.3.7 python-blosc=1.8.1 pathos graphviz python-graphviz rise textblob wordcloud quandl pyspark py4j ^\n",
    "spacy fastparquet python-snappy nodejs koalas textacy pystan fbprophet --yes\n",
    "\n",
    "call pip install arctic==1.79.2\n",
    "\n",
    "REM install tensorflow, transformers/huggingface and table readers\n",
    "REM only if you have GPU\n",
    "REM call pip install tensorflow-gpu transformers cvlib pytesseract tabula-py\n",
    "call pip install tensorflow==2.1.0 transformers pytesseract tabula-py\n",
    "\n",
    "REM install PyTorch\n",
    "REM only if you have GPU\n",
    "REM call conda install pytorch torchvision cudatoolkit=10.0 -c pytorch --yes\n",
    "call conda install pytorch torchvision cpuonly -c pytorch --yes\n",
    "call conda install -c conda-forge opencv --yes\n",
    "call pip install cvlib\n",
    "\n",
    "REM for findatapy (and NLP)\n",
    "call pip install fxcmpy alpha_vantage yfinance twython newspaper3k seasonal pdfminer.six vaderSentiment gensim\n",
    "\n",
    "REM findatapy, chartpy and finmarketpy\n",
    "REM needs git\n",
    "REM call pip install git+https://github.com/cuemacro/finmarketpy.git git+https://github.com/cuemacro/chartpy.git git+https://github.com/cuemacro/findatapy.git\n",
    "call pip install finmarketpy chartpy findatapy\n",
    "\n",
    "REM graphics libraries\n",
    "call pip install cufflinks==0.17 plotly_express==0.4.1 dash==1.8.0 dash-html-components==1.0.2 dash-core-components==1.7.0 plotly==4.5.0 dash-table==4.6.0\n",
    "\n",
    "REM to be able to plot Plotly into PNG or JPG\n",
    "call conda install -c plotly plotly-orca --yes\n",
    "\n",
    "REM in case has been installed elsewhere by pip\n",
    "call pip uninstall pandas\n",
    "\n",
    "REM newer versions of Pandas aren't supported as well by other libraries (eg. Arctic), so stick to 0.24.2\n",
    "call conda install -c anaconda pandas=0.24.2 pyarrow=0.13.0 --yes\n",
    "\n",
    "REM Jupyter libraries\n",
    "call jupyter contrib nbextension install --user\n",
    "call jupyter nbextension enable execute_time/ExecuteTime\n",
    "call jupyter-nbextension install rise --py --sys-prefix\n",
    "call jupyter labextension install @jupyter-widgets/jupyterlab-manager@1.1 --no-build\n",
    "call jupyter labextension install plotlywidget@1.4.0 --no-build\n",
    "call jupyter labextension install jupyterlab-plotly@1.4.0 --no-build\n",
    "call jupyter lab build\n",
    "\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing a conda environment for Windows (alternative method 2 - quicker!)\n",
    "\n",
    "Alternatively we can download [environment_windows.yml](environment_windows.yml) and use that to create our conda environment with all the necessary packages.\n",
    "\n",
    "* Open up the Anaconda Prompt (should be in the Start Menu)\n",
    "* In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in `conda env create -f environment_windows.yml` and press Enter\n",
    "    * Anaconda will then run all the necessary command to download the various packages above using conda and pip\n",
    "    * At the end of the process you'll have a Python 3.6 conda environment with the name 'py36class' which you can use, with all the packages you'll need for this course, such as pandas, numpy etc.\n",
    "    * Note that you may need to put in the full path for wherever you downloaded the environment_windows.yml file\n",
    "    \n",
    "* The `environment.yml` file (or similar name) basically has all the instructions required to recreate a conda environment\n",
    "* To create your own `environment.yml` file (for backup purposes, or if you'd like to distribute your conda environment) run the below command in your Anaconda Prompt\n",
    "    * `conda env export > environment.yml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing xlwings addin in Excel on Windows\n",
    "\n",
    "Both of the above should install the xlwings Python library. You also need to add the xlwings addin to Excel too. Instructions for this can be found at https://docs.xlwings.org/en/stable/addin.html (xlwings is also supported on Excel for Mac, although the functionality may differ. It won't work on Linux, given there's no Linux version of Excel)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:12:06.412884Z",
     "start_time": "2019-10-01T16:12:06.404503Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Installing a conda environment for Linux and Mac\n",
    "\n",
    "A conda environment is a separate version of Python, where you can install all your own Python environments. For Linux \n",
    "and Mac, we'll install more libraries, which you might need to use later (some of which aren't fully supported by Windows).\n",
    "\n",
    "* Download \n",
    "* Open up a text editor\n",
    "* Copy the below text\n",
    "* Create a folder `/Users/<your-username>/pythoncourse` for Mac or `/home/<your-username>/pythoncourse` for Linux\n",
    "* Save the file as `create_conda_env.sh` in the pythoncourse folder\n",
    "* Open a Terminal window (usually a black window icon on both Linux and Mac)\n",
    "* Type in (or wherever you installed Anaconda) and press enter to run\n",
    "    * `cd /Users/<your-username>/anaconda3/bin` for Mac\n",
    "    * `cd /home/<your-username>/anaconda3/bin` for Linux\n",
    "* conda is in this folder\n",
    "* Note, conda might already be on your path - you can check this by typing `conda` and pressing enter, to see if you get an error. On Windows\n",
    "the installer will have a default option to add Anaconda to your path\n",
    "* On Linux/Mac, it is recommend to add the Anaconda folder to your path\n",
    "* You can temporarily add paths in Linux/Mac using the following command in the terminal (change the path to wherever you installed\n",
    "Anaconda)\n",
    "    * `export PATH=/Users/<your-username>/anaconda3/bin:$PATH` for Mac\n",
    "    * `export PATH=/home/<your-username>/anaconda3/bin:$PATH` for Linux\n",
    "* To permanently add folders to your path by editing a file like `.bashrc` which is in your home folder\n",
    "    * http:https://osxdaily.com/2015/07/28/set-enviornment-variables-mac-os-x/ for Mac\n",
    "    * https://opensource.com/article/17/6/set-path-linux for Linux\n",
    "* Then run\n",
    "    * `./Users/<your-username>/pythoncourse/create_conda_env.sh` for Mac\n",
    "    * `./home/<your-username>/pythoncourse/create_conda_env.sh` for Linux\n",
    "* It will likely take a long time, and you might need to periodically click yes also Windows to change settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:13:50.842110Z",
     "start_time": "2019-10-01T16:13:50.828861Z"
    },
    "pycharm": {}
   },
   "source": [
    "### Copy the below to create_conda_env.sh (or [download it](create_conda_env.sh))\n",
    "\n",
    "```\n",
    "#!/usr/bin/env bash\n",
    "\n",
    "# update conda\n",
    "conda update -n base -c defaults conda --yes\n",
    "\n",
    "# conda config --set allow_conda_downgrades true\n",
    "# conda install conda=4.6.11\n",
    "\n",
    "source activate\n",
    "\n",
    "# remove any existing environment called py36class, and create a py36class with anaconda packages\n",
    "conda remove -n py36class --all --yes\n",
    "conda create -n py36class python=3.6 anaconda\n",
    "source activate py36class\n",
    "\n",
    "# install from conda-forge (pyarrow is a newer version!)\n",
    "conda install -c conda-forge \\\n",
    "jupyterlab jupyter_contrib_nbextensions jupyter_nbextensions_configurator \\\n",
    "redis-py=3.3.7 python-blosc=1.8.1 pathos graphviz python-graphviz rise textblob wordcloud quandl \\\n",
    "vaex-core vaex-viz vaex-jupyter vaex-arrow vaex-server vaex-hdf5 vaex-astro vaex-distributed pyspark py4j \\\n",
    "spacy fastparquet python-snappy nodejs koalas textacy pystan fbprophet --yes\n",
    "\n",
    "# for database\n",
    "pip install arctic==1.79.2\n",
    "\n",
    "# celery library\n",
    "pip install celery==4.3.0 celery[redis] celery[msgpack] msgpack-python\n",
    "\n",
    "# for findatapy (and NLP)\n",
    "pip install fxcmpy alpha_vantage yfinance twython newspaper3k seasonal pdfminer.six vaderSentiment gensim\n",
    "\n",
    "# install tensorflow and transformers/huggingface (on Mac OS can have issues installing TensorFlow, so if you have Mac\n",
    "# comment it out)\n",
    "# only if you have GPU\n",
    "# pip install tensorflow-gpu transformers cvlib pytesseract tabula-py\n",
    "pip install tensorflow==2.1.0 transformers pytesseract tabula-py\n",
    "\n",
    "# install PyTorch\n",
    "# only if you have GPU\n",
    "# conda install pytorch torchvision cudatoolkit=10.0 -c pytorch --yes\n",
    "conda install pytorch torchvision cpuonly -c pytorch --yes\n",
    "conda install -c conda-forge opencv --yes\n",
    "pip install cvlib\n",
    "\n",
    "# modin (later versions need later versions of pandas)\n",
    "# may downgrade to pandas to 0.24.2\n",
    "pip install modin=0.5.0\n",
    "\n",
    "# findatapy, chartpy and finmarketpy (or can manually clone these from GitHub and add to your Python path)\n",
    "# need to have Git\n",
    "# pip install git+https://github.com/cuemacro/finmarketpy.git git+https://github.com/cuemacro/chartpy.git git+https://github.com/cuemacro/findatapy.git\n",
    "pip install finmarketpy chartpy findatapy\n",
    "\n",
    "# graphics libraries\n",
    "pip install cufflinks==0.17 plotly_express==0.4.1 dash==1.8.0 dash-html-components==1.0.2 dash-core-components==1.7.0 plotly==4.5.0 dash-table==4.6.0\n",
    "\n",
    "# to be able to plot Plotly into PNG or JPG\n",
    "conda install -c plotly plotly-orca --yes\n",
    "\n",
    "# in case has been installed elsewhere\n",
    "pip uninstall pandas\n",
    "\n",
    "# newer versions of Pandas aren't supported as well by other libraries (eg. Arctic), so stick to 0.24.2\n",
    "conda install -c anaconda pandas=0.24.2 pyarrow=0.13.0 --yes\n",
    "\n",
    "# GPU libraries\n",
    "# conda install -c rapidsai -c nvidia -c conda-forge -c defaults rapids=0.10 python=3.6\n",
    "\n",
    "# in case modin has changed pandas version!\n",
    "# conda install -c anaconda pandas=0.24.2 --yes\n",
    "\n",
    "# Jupyter and Jupyterlab extensions\n",
    "jupyter contrib nbextension install --user # to activate js on Jupyter\n",
    "jupyter nbextension enable execute_time/ExecuteTime\n",
    "jupyter-nbextension install rise --py --sys-prefix\n",
    "jupyter labextension install @jupyter-widgets/jupyterlab-manager@1.1 --no-build\n",
    "jupyter labextension install plotlywidget@1.4.0 --no-build\n",
    "jupyter labextension install jupyterlab-plotly@1.4.0 --no-build\n",
    "jupyter lab build\n",
    "\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing a conda environment for Linux (alternative method 2 - quicker!)\n",
    "\n",
    "Alternatively we can download [environment_linux.yml](environment_linux.yml) and use that to create our conda environment with all the necessary packages. This should be a lot faster to run.\n",
    "\n",
    "* Open up a terminal window (making sure that Anaconda is on your path)\n",
    "* In this prompt, your Anaconda folder will be on the path (ie. it will recognise where `conda` is installed etc.)\n",
    "    * Type in `conda env create -f environment_linux.yml` and press Enter\n",
    "    * Anaconda will then run all the necessary command to download the various packages above using conda and pip\n",
    "    * At the end of the process you'll have a Python 3.6 conda environment with the name 'py36class' which you can use, with all the packages you'll need for this course, such as pandas, numpy etc.\n",
    "    * Note that you may need to put in the full path for wherever you downloaded the environment_windows.yml file\n",
    "    \n",
    "* The `environment.yml` file (or similar name) basically has all the instructions required to recreate a conda environment\n",
    "* To create your own `environment.yml` file (for backup purposes, or if you'd like to distribute your conda environment) run the below command in your Anaconda Prompt\n",
    "    * `conda env export > environment.yml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-01T16:04:01.263522Z",
     "start_time": "2019-10-01T16:04:01.258640Z"
    },
    "pycharm": {}
   },
   "source": [
    "## Download and install PyCharm\n",
    "\n",
    "* If you running Python locally, you can try to install PyCharm, which makes it easier to develop Python code\n",
    "* You can download and install PyCharm Community from https://www.jetbrains.com/pycharm/download\n",
    "* There are versions for PyCharm for Windows, Linux and Mac operating systems.\n",
    "* PyCharm IDE makes it easier to write and run Python code\n",
    "* In File / Settings - you will likely need to set PyCharm to use your py36class environment, which is likely at the following locations\n",
    "    * Windows: `C:\\Users\\<your-username>\\Anaconda3\\envs\\py36class`\n",
    "    * macOS: `/Users/<your-username>/anaconda3/envs/py36class`\n",
    "    * Linux: `/home/<your-username>/anaconda3/envs/py36class`\n",
    "* PyCharm will create a workspace for you, where you can place your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Microsoft Azure Notebooks with Jupyter\n",
    "\n",
    "If you do not want to install Anaconda on your own machine (or if you don't have the correct permissions to do so), you can instead use Microsoft Azure Notebooks, which gives you a Jupyter notebook in the cloud. This can be a good solution for those wanting to learn Python. \n",
    "\n",
    "### Creating your first Jupyter notebook in Microsoft Azure Notebooks\n",
    "\n",
    "Below we outline the steps to creating your first Jupyter notebook in Microsoft Azure Notebooks.\n",
    "\n",
    "* Go to https://notebooks.azure.com/ and sign in with your Microsoft account\n",
    "    * If you do not already have a Microsoft account eg. for Outlook.com you can sign up for a free account or on the Azure notebook website\n",
    "* At https://notebooks.azure.com/TYPEYOURUSERNAME/projects you will be able to see the projects you currently have\n",
    "* Create a new project eg. named `pythoncourse` in which to place all your Jupyter notebooks\n",
    "\n",
    "### Creating your conda environment in Microsoft Azure Notebooks (harder)\n",
    "\n",
    "* You'll also need to set this new project to use the conda environment ready for the course, so it has all the right Python libraries ready for you to use\n",
    "    * At https://docs.microsoft.com/en-us/azure/notebooks/quickstart-create-jupyter-notebook-project-environment you can find additional details on how to create a conda environment for Azure Notebooks (using an environment.yml file)\n",
    "    * You'll need the [environment_azure_linux.yml](environment_azure_linux.yml) and then upload it to your newly made project\n",
    "    * Note, some libraries may not work in Azure notebooks such as the Bloomberg Python API\n",
    "    * GPU versions of some of the libraries may not work, unless you set to Azure notebooks to use a (paid-for) GPU enabled workstation\n",
    "* There are other similar solutions for Jupyter in the cloud, eg. Google Colab etc. which you might also consider\n",
    "\n",
    "### Alternatively in your Azure Jupyter notebook to setup your Python environment (easier)\n",
    "\n",
    "* If you're having trouble getting Azure to setup your conda environment with the above method (often it isn't clear when Azure has setup the conda environment from above). In that case, it is suggested to do the below, which will also execute a lot quicker\n",
    "* Here we are trying to only install some of the most important additional packages, as Azure already comes with many to begin, and this will reduce the download/install time\n",
    "* We've also excluded some which are likely to are less stable in the Azure environment (or require a lot of other dependencies) \n",
    "* We've rewritten to use pip, as conda can take a while on slower machines (except for packages which are difficult to install with pip like plotly-orca, which is optional)\n",
    "* Then run the below commands in your Jupyter notebook (this will likely take a long time, usually a couple of minutes)\n",
    "* Make sure you are running Azure notebooks in Python 3.6 (not 3.5)\n",
    "* Note, that you might to run these every so often, because after 60 minutes of inactivity Azure will kill your notebook instance (and hence any Python library installations you do will also disappear)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A selection of libraries (note: this isn't everything)\n",
    "!pip install \\\n",
    "  redis==3.3.7 pathos pyarrow==0.13.0 pandas==0.24.2 wordcloud quandl \\\n",
    "  vaex-core vaex-viz vaex-jupyter vaex-server vaex-hdf5 vaex-astro vaex-distributed pyspark py4j \\\n",
    "  spacy celery==4.3.0 celery[redis] celery[msgpack] msgpack-python \\\n",
    "  fxcmpy alpha_vantage yfinance twython newspaper3k seasonal pdfminer.six vaderSentiment gensim textacy \\\n",
    "  transformers \\\n",
    "  finmarketpy chartpy findatapy \\\n",
    "  cufflinks==0.17 plotly_express==0.4.1 \\\n",
    "  plotly==4.2.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also try installing plotly-orca via conda (this is useful if you want to output plotly charts as PNG, but is optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!conda install -c plotly plotly-orca --yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or rather than running the above commands, you can try creating a full conda environment in Azure from the yml file (although from experience this is less likely to work, is much slower and it's difficult to activate later)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!conda env create -f environment_azure_linux.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check if your Azure Python package installation has worked by importing Pandas and checking the version is indeed 0.24.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-20T12:35:44.464099Z",
     "start_time": "2020-02-20T12:35:43.650407Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.24.2'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas\n",
    "pandas.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Direct compute on a paid Azure Jupyter instance (Data Science Virtual Machine)\n",
    "\n",
    "The free version of Azure notebooks has a number of limitations. In particular, that it resets every hour and only has 4GB RAM. You can instead create paid instances Data Science Virtual Machine for Linux (Ubuntu) on Azure (DSVM), which is a Linux box for a number of different CPUs/RAM which are much more powerful. You can configure the machine as want, and the configuration will be stored for as long as you have the instance. Do make sure to power down the machine, once you are finished using it. This won't wipe the hard disk, but it will mean you aren't paying for it when it's idle.\n",
    "\n",
    "Whilst Azure 3.6 ML from the DSVM already has many libraries installed, eg. TensorFlow etc. You will still need to install additional Python libraries, but this can be done once. Script `update_azure_conda_env.sh` will attempt to install these extra libraries such as NLTK etc. and also a version of Pandas which should be more compatible.\n",
    "\n",
    "You can then connect your Azure notebook to run on this server via \"Direct compute\", where you need to specify the IP of the machine. At https://docs.microsoft.com/en-us/azure/notebooks/use-data-science-virtual-machine it gives details on how to do \"Direct compute\" on your Azure Notebook. If you get a powerful machine you should be able to host multiple users. However, you should add usernames/passwords for every user if you do this, so each user runs in their own instance.\n",
    "\n",
    "You may need to run `conda install ipykernel` to make a new conda environment appear in the Jupyter notebook. Make sure that you select the right Python kernel from the Jupyter menu (Kernel / Change Kernel)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test your Python installation by running this.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can test your Python installation, by starting the Anaconda prompt (switching to the right conda environment) and then starting Jupyter (in Azure Notebook, you'll start a Jupyter notebook via it's own interface) using the below commands. \n",
    "\n",
    "Note, that you will likely have to change the default notebook-dir parameter (or you can just omit it, in which case Jupyter will use the current working directory)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    conda activate py36class\n",
    "    jupyter notebook --notebook-dir='e:/Remote/pythoncourse/pythoncourse/notebooks'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then try running the below Python code in a Jupyter notebook or the Python interpreter to see if some of the libraries we've installed work. This is not an exhaustive test, but only a few which we'll use a lot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-04T18:38:51.129577Z",
     "start_time": "2019-12-04T18:38:51.123720Z"
    }
   },
   "outputs": [],
   "source": [
    "import chartpy\n",
    "import quandl\n",
    "import finmarketpy\n",
    "import findatapy\n",
    "import pandas\n",
    "import numpy\n",
    "import plotly"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}