Lab_-_M4L4b_-_Hierarchical_Clustering_-_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.bigdatauniversity.com\"><img src = \"https://ibm.box.com/shared/static/wbqvbi6o6ip0vz55ua5gp17g4f1k7ve9.png\" width = 300, align = \"center\"></a>\n",
    "# <center>Hierarchical Clustering</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <b>Welcome to Lab 4b of Machine Learning with Python.</b>\n",
    "<p><b>Machine Learning is a form of artificial intelligence (AI), where the system can \"learn\" without explicitly being coded</b></p>\n",
    "\n",
    "In this lab exercise, you will learn some popular machine learning algorithms. For <b>supervised learning</b>, we will discuss <b>decision trees</b> and <b>random forests</b>. In <b>unsupervised learning</b>, we will discuss <b>k-means clustering</b>, <b>agglomerative hierarchical clustering</b>, and <b>density-based clustering</b> or <b>DBSCAN</b>.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some Notebook Commands Reminders:\n",
    "<ul>\n",
    "    <li>Run a cell: CTRL + ENTER</li>\n",
    "    <li>Create a cell above a cell: a</li>\n",
    "    <li>Create a cell below a cell: b</li>\n",
    "    <li>Change a cell to Markdown: m</li>\n",
    "    \n",
    "    <li>Change a cell to code: y</li>\n",
    "</ul>\n",
    "\n",
    "<b> If you are interested in more keyboard shortcuts, go to Help -> Keyboard Shortcuts </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> <i> Before starting the lab, please run the following code in order to access the solutions </i> </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.core.display import HTML\n",
    "HTML(\"\"\"\n",
    "<style type=\"text/css\">\n",
    "    #ans:hover { background-color: black; }\n",
    "    #ans {padding: 6px; \n",
    "        background-color: white; \n",
    "        border: green 2px solid;\n",
    "        font-weight: bold}\n",
    "</style>\n",
    "\"\"\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <u> Hierarchical Clustering - Agglomerative</u> \n",
    "\n",
    "We will be looking at the next clustering technique, which is <b>Agglomerative Hierarchical Clustering</b>. Remember that agglomerative is the bottom up approach. <br> <br>\n",
    "In this lab, we will be looking at Agglomerative clustering, which is more popular than Divisive clustering. <br> <br>\n",
    "We will also be using Complete Linkage as the Linkage Criteria. <br>\n",
    "<b> <i> NOTE: You can also try using Average Linkage wherever Complete Linkage would be used to see the difference! </i> </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Import Libraries:\n",
    "<ul>\n",
    "    <li> <b>numpy as np</b> </li>\n",
    "    <li> <b>ndimage</b> from <b>scipy</b> </li>\n",
    "    <li> <b>hierarchy</b> from <b>scipy.cluster</b> </li>\n",
    "    <li> <b>pyplot as plt</b> from <b>matplotlib</b> </li>\n",
    "    <li> <b>manifold</b> from <b>sklearn</b> </li>\n",
    "    <li> <b>datasets</b> from <b>sklearn</b> </li>\n",
    "    <li> <b>AgglomerativeClustering</b> from <b>sklearn</b> </li>\n",
    "    <li> <b>make_blobs</b> from <b>sklearn.datasets.samples_generator</b> </li>\n",
    "</ul> <br>\n",
    "Also run <b>%matplotlib inline</b> that that wasn't run already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "from scipy import ndimage \n",
    "from scipy.cluster import hierarchy \n",
    "from scipy.spatial import distance_matrix \n",
    "from matplotlib import pyplot as plt \n",
    "from sklearn import manifold, datasets \n",
    "from sklearn.cluster import AgglomerativeClustering \n",
    "from sklearn.datasets.samples_generator import make_blobs \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hover over the green box below for the answer\n",
    "<p id = \"ans\"> <font color = \"white\"> \n",
    "import numpy as np <br>\n",
    "from scipy import ndimage <br>\n",
    "from scipy.cluster import hierarchy <br>\n",
    "from scipy.spatial import distance_matrix <br>\n",
    "from matplotlib import pyplot as plt <br>\n",
    "from sklearn import manifold, datasets <br>\n",
    "from sklearn.cluster import AgglomerativeClustering <br>\n",
    "from sklearn.datasets.samples_generator import make_blobs <br>\n",
    "%matplotlib inline\n",
    "</font> </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Generating Random Data\n",
    "We will be generating another set of data using the <b>make_blobs</b> class once again. This time you will input your own values! <br> <br>\n",
    "Input these parameters into make_blobs:\n",
    "<ul>\n",
    "    <li> <b>n_samples</b>: The total number of points equally divided among clusters. </li>\n",
    "    <ul> <li> Choose a number from 10-1500 </li> </ul>\n",
    "    <li> <b>centers</b>: The number of centers to generate, or the fixed center locations. </li>\n",
    "    <ul> <li> Choose arrays of x,y coordinates for generating the centers. Have 1-10 centers (ex. centers=[[1,1], [2,5]]) </li> </ul>\n",
    "    <li> <b>cluster_std</b>: The standard deviation of the clusters. The larger the number, the further apart the clusters</li>\n",
    "    <ul> <li> Choose a number between 0.5-1.5 </li> </ul>\n",
    "</ul> <br>\n",
    "Save the result to <b>X2</b> and <b>y2</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X2, y2 = make_blobs(n_samples=50, centers=[[4,4], [-2, -1], [1, 1], [10,4]], cluster_std=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hover over the green box below for an example answer (This will be used for the remainder of the example).\n",
    "<p id = \"ans\"> <font color = \"white\"> \n",
    "X2, y2 = make_blobs(n_samples=50, centers=[[4,4], [-2, -1], [1, 1], [10,4]], cluster_std=0.9)\n",
    "</font> </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the scatter plot of the randomly generated data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.scatter(X2[:, 0], X2[:, 1], marker='.') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Agglomerative Clustering\n",
    "We will start by clustering the random data points we just created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <b> AgglomerativeClustering </b> class will require two inputs:\n",
    "<ul>\n",
    "    <li> <b>n_clusters</b>: The number of clusters to form as well as the number of centroids to generate. </li>\n",
    "    <ul> <li> Value will be: 4 </li> </ul>\n",
    "    <li> <b>linkage</b>: Which linkage criterion to use. The linkage criterion determines which distance to use between sets of observation. The algorithm will merge the pairs of cluster that minimize this criterion. </li>\n",
    "    <ul> \n",
    "        <li> Value will be: 'complete' </li> \n",
    "        <li> <b>Note</b>: It is recommended you try everything with 'average' as well </li>\n",
    "    </ul>\n",
    "</ul> <br>\n",
    "Save the result to a variable called <b> agglom </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "agglom = AgglomerativeClustering(n_clusters = 4, linkage = 'average')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hover over the green box below for the answer\n",
    "<p id = \"ans\"> <font color = \"white\"> \n",
    "agglom = AgglomerativeClustering(n_clusters = 4, linkage = 'average')\n",
    "</font> </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the model with <b> X2 </b> and <b> y2 </b> from the generated data above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "agglom.fit(X2,y2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hover over the green box below for the answer\n",
    "<p id = \"ans\"> <font color = \"white\"> \n",
    "agglom.fit(X2,y2)\n",
    "</font> </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following code to show the clustering! <br>\n",
    "Remember to read the code and comments to gain more understanding on how the plotting works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a figure of size 6 inches by 4 inches.\n",
    "plt.figure(figsize=(6,4))\n",
    "\n",
    "# These two lines of code are used to scale the data points down,\n",
    "# Or else the data points will be scattered very far apart.\n",
    "\n",
    "# Create a minimum and maximum range of X2.\n",
    "x_min, x_max = np.min(X2, axis=0), np.max(X2, axis=0)\n",
    "\n",
    "# Get the average distance for X2.\n",
    "X2 = (X2 - x_min) / (x_max - x_min)\n",
    "\n",
    "# This loop displays all of the datapoints.\n",
    "for i in range(X2.shape[0]):\n",
    "    # Replace the data points with their respective cluster value \n",
    "    # (ex. 0) and is color coded with a colormap (plt.cm.spectral)\n",
    "    plt.text(X2[i, 0], X2[i, 1], str(y2[i]),\n",
    "             color=plt.cm.spectral(agglom.labels_[i] / 10.),\n",
    "             fontdict={'weight': 'bold', 'size': 9})\n",
    "    \n",
    "# Remove the x ticks, y ticks, x and y axis\n",
    "plt.xticks([])\n",
    "plt.yticks([])\n",
    "plt.axis('off')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()\n",
    "\n",
    "# Display the plot of the original data before clustering\n",
    "plt.scatter(X2[:, 0], X2[:, 1], marker='.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Dendrogram Associated for the Agglomerative Hierarchical Clustering\n",
    "Remember that a <b>distance matrix</b> contains the <b> distance from each point to every other point of a dataset </b>. <br>\n",
    "Use the function <b> distance_matrix, </b> which requires <b>two inputs</b>. Use the Feature Matrix, <b> X2 </b> as both inputs and save the distance matrix to a variable called <b> dist_matrix </b> <br> <br>\n",
    "Remember that the distance values are symmetric, with a diagonal of 0's. This is one way of making sure your matrix is correct. <br> (print out dist_matrix to make sure it's correct)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dist_matrix = distance_matrix(X2,X2) \n",
    "print(dist_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hover over the green box below for the answer\n",
    "<p id = \"ans\"> <font color = \"white\"> \n",
    "dist_matrix = distance_matrix(X2,X2) <br>\n",
    "print(dist_matrix)\n",
    "</font> </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the <b> linkage </b> class from hierarchy, pass in the parameters:\n",
    "<ul>\n",
    "    <li> The distance matrix </li>\n",
    "    <li> 'complete' for complete linkage </li>\n",
    "</ul> <br>\n",
    "Save the result to a variable called <b> Z </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Z = hierarchy.linkage(dist_matrix, 'complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Hover over the green box below for the answer\n",
    "<p id = \"ans\"> <font color = \"white\"> \n",
    "Z = hierarchy.linkage(dist_matrix, 'complete')\n",
    "</font> </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Next, we will save the dendrogram to a variable called <b>dendro</b>. In doing this, the dendrogram will also be displayed.\n",
    "Using the <b> dendrogram </b> class from hierarchy, pass in the parameter:\n",
    "<ul> <li> Z </li> </ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dendro = hierarchy.dendrogram(Z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Hover over the green box below the answer\n",
    "<p id = \"ans\"> <font color = \"white\"> \n",
    "dendro = hierarchy.dendrogram(Z)\n",
    "</font> </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Want to learn more?\n",
    "\n",
    "IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: [SPSS Modeler for Mac users](https://cocl.us/ML0101EN_SPSSMod_mac) and [SPSS Modeler for Windows users](https://cocl.us/ML0101EN_SPSSMod_win)\n",
    "\n",
    "Also, you can use Data Science Experience to run these notebooks faster with bigger datasets. Data Science Experience is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, DSX enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of DSX users today with a free account at [Data Science Experience](https://cocl.us/ML0101EN_DSX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Additional Resources\n",
    "<br>\n",
    "General Clustering: http:https://scikit-learn.org/stable/modules/clustering.html \n",
    "<br><br>\n",
    "Hierarchical Clustering: <br>\n",
    "https://www.youtube.com/watch?v=OcoE7JlbXvY <br>\n",
    "https://www.youtube.com/watch?v=MIWVfCcHzM4 <br>\n",
    "https://www.youtube.com/watch?v=2z5wwyv0Zk4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "Copyright &copy; 2016 [Big Data University](https://bigdatauniversity.com/?utm_source=bducopyrightlink&utm_medium=dswb&utm_campaign=bdu). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/).​"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}