09_Model_Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class 11 Agenda:\n",
    "  * **Null accuracy**\n",
    "  * **Confusion matrix**\n",
    "  * **sensitivity, specificity, accuracy**\n",
    "  * **ROC curves, AUC, setting a threshold**\n",
    "  * **ROC curves, AUC: All that matters is that ordering is preserved**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today we are going to talk about how to more accurately measure models for supervised classification. Because supervised classification problems are ubiquitous (will this person churn? will this ad be clicked? will this stock go up tomorrow?), being able to evaluate how well a supervised classifier works and to be able to choose what defines a \"well-performing\" classifier is very important.\n",
    "\n",
    "To that end, we will also learn how to:\n",
    "  * decompose the kinds of errors a trained model makes (on unseen data)\n",
    "  * decide where (at what probability) to threshold a binary classifier (one that decides between two choices) given what is acceptable in terms of the kinds of errors the model is expected to make in the wild. \n",
    "  * understand confusion matrices\n",
    "  \n",
    "** By the end of this notebook you will:**\n",
    "\n",
    "- Have a working conceptual understanding of key aspects of model evaluation in machine learning\n",
    "- Be able to interpret key model metrics computed using scikit-learn\n",
    "- Be able to to use different model metrics for model evaluation, depending on the goals of your model\n",
    "- Be able to explain what an ROC curve and AUC metric are and how they should be used when evaluating classifiers and setting classifier thresholds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import everything we're going to use today, like always:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data handling, model creation/evaluation\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import KFold, train_test_split, cross_val_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor\n",
    "from sklearn.preprocessing import LabelEncoder, StandardScaler, PolynomialFeatures\n",
    "from sklearn import metrics\n",
    "import scipy.stats as stats\n",
    "\n",
    "# visualization\n",
    "%matplotlib inline\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Null Accuracy\n",
    "\n",
    "Null accuracy measures what our expected accuracy should be if we were to **use the most frequent response (most frequent class) as our prediction for every new sample we see.** It is a baseline against which you may want to measure your classifier, especially when your classes are really unbalanced.\n",
    "\n",
    "In class 5, when we learned about [Logistic Regression](https://en.wikipedia.org/wiki/Logistic_regression), we attempted to build a predictive model on a dataset that was fairly unbalanced (the vertebral column dataset from UCI).\n",
    "\n",
    "Let's use that dataset again to measure null accuracy, and see whether a basic Logistic Regression model can give higher test set accuracy than this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set accuracy of LR model:  0.8709677419354839\n"
     ]
    }
   ],
   "source": [
    "vertebral_data = pd.read_csv(\"./data/vertebral_column_2_categories.dat\",sep=\" \", \n",
    "                             names=[\"pelvic_incidence\",\"pelvic_tilt\",\"lumbar_lordosis_angle\",\n",
    "                                    \"sacral_slope\",\"pelvic_radius\",\"spondy_grade\",\"outcome\"])\n",
    "vertebral_data.outcome.value_counts()\n",
    "vertebral_data.outcome = (vertebral_data.outcome ==\"AB\").astype(int)\n",
    "X = vertebral_data[vertebral_data.columns.tolist()[:-1]]\n",
    "y = vertebral_data.outcome\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,random_state=1)\n",
    "\n",
    "lr = LogisticRegression(C=1e9,solver='lbfgs')\n",
    "lr.fit(X_train,y_train)\n",
    "y_test_pred = lr.predict(X_test)\n",
    "\n",
    "print(\"Test set accuracy of LR model: \",metrics.accuracy_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is the null accuracy here?**\n",
    "\n",
    "It is simply the fraction of `AB` (class 1) samples in the test set (since that is the most frequent class in the whole dataset):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Null accuracy on the test set:  0.7096774193548387\n"
     ]
    }
   ],
   "source": [
    "# compute null accuracy manually\n",
    "print(\"Null accuracy on the test set: \",y_test.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute this using the scikit-learn api by creating what's called a `DummyClassifier`.\n",
    "\n",
    "It can create a variety of dummy models based on simple statistics it is trained on.\n",
    "\n",
    "We are going to create one where the model simply predicts the most frequent class, by passing in a `string` to the `strategy` parameter when we generate the unfitted model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most frequent class dummy classifier test accuracy:  0.7096774193548387\n"
     ]
    }
   ],
   "source": [
    "from sklearn.dummy import DummyClassifier\n",
    "dumb_model = DummyClassifier(strategy='most_frequent')\n",
    "dumb_model.fit(X_train, y_train)\n",
    "y_dumb_class = dumb_model.predict(X_test)\n",
    "print(\"Most frequent class dummy classifier test accuracy: \",metrics.accuracy_score(y_test, y_dumb_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise Time\n",
    "\n",
    "* Use the same dummy classifier approach and train an LR model using 10-fold cross validation and compute the accuracy score of each model.\n",
    "* Do the same with 30-fold cross validation and plot both test-set accuracy distributions. Are the two distributions overlapping? If they aren't what does that mean?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix\n",
    "\n",
    "In order to more fully understand the kinds of mistakes the model is making, we need to investigate what is called the **confusion matrix** of the model on unseen (test) data. The confusion matrix simply counts the number of predictions that fall into each possible prediction bucket.\n",
    "\n",
    "So, for a 2-class classification problem, there are 4 \"prediction buckets\":\n",
    "  * predict 0 (normal), actual 0 (normal) - called a **correct rejection/true negative**\n",
    "  * predict 0 (normal), actual 1 (abnormal) <-- this is an error called a **miss/false negative**\n",
    "  * predict 1 (abnormal), actual 0 (normal) <-- this is an error called a **false alarm/false positive**\n",
    "  * predict 1 (abnormal), actual 1 (abnormal) - called a **hit/true positive**\n",
    "  \n",
    "![confusion matrix](./images/confusion_matrix.png)\n",
    "\n",
    "Let's compute the confusion matrix on the test set for our Logistic Regression model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[21,  6],\n",
       "       [ 6, 60]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# confusion matrix\n",
    "cm = metrics.confusion_matrix(y_test, y_test_pred)\n",
    "cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By convention in a confusion matrix, the actual categories are the rows and the predicted values are the columns. Our confusion matrix shows that on the test set we have an equal number of misses and false alarms (both values are 6).\n",
    "\n",
    "We can break down the confusion matrix into a variety of single-value metrics that answer specific questions about how our model is expected to fare on new data:\n",
    "\n",
    "  * **sensitivity/true positive rate(TPR)/recall:** What fraction of the \"abnormal\" samples in unseen data did we correctly predict? \n",
    "  $$ TPR = \\frac {\\sum TP}{\\sum (TP+FN)}$$\n",
    "  * **specificity/true negative rate(TNR):** What fraction of \"normal\" samples in unseen data did we correctly predict?  \n",
    "  $$ TNR = \\frac {\\sum TN}{\\sum (TN+FP)}$$\n",
    "  * **precision/positive predictive value(PPV)** How frequently is our model correct when it predicts \"abnormal\" on new data?\n",
    "  $$ PPV = \\frac {\\sum TP}{\\sum (TP+FP)}$$\n",
    "  * **negative predictive value (NPV):** How frequently is our model correct when it predicts \"normal\" on new data? \n",
    "  $$ NPV = \\frac {\\sum TN}{\\sum (TN+FN)}$$\n",
    "  * **accuracy (ACC):** How frequently is our model correct on all new data, regardless of class?\n",
    "  $$ ACC = \\frac {\\sum (TN+TP)}{\\sum (TN+FN+TP+FP)}$$\n",
    "  * **F1 score (F1): ** The harmonic mean of precision and recall:\n",
    "  $$ F1 = 2*\\frac {Precision*Recall}{Precision+Recall}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity/Recall (TPR):  0.9090909090909091\n",
      "Specificity (TNR):  0.7777777777777778\n",
      "Precision (PPV):  0.9090909090909091\n",
      "NPV:  0.7777777777777778\n",
      "Accuracy:  0.8709677419354839\n",
      "F1: 0.9090909090909091\n"
     ]
    }
   ],
   "source": [
    "# calculate each metric by hand\n",
    "print(\"Sensitivity/Recall (TPR): \",cm[1,1] / float(cm[1,1] + cm[1,0]))\n",
    "print(\"Specificity (TNR): \", cm[0,0] / float(cm[0,0] + cm[0,1]))\n",
    "print(\"Precision (PPV): \", cm[1,1] / float(cm[1,1]+cm[0,1]))\n",
    "print(\"NPV: \", cm[0,0] / float(cm[0,0]+cm[1,0]))\n",
    "print(\"Accuracy: \", (cm[1,1]+cm[0,0]) / float(cm.sum()))\n",
    "print(\"F1:\", metrics.f1_score(y_test,y_test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sensitivity/Recall (TPR):  0.9090909090909091\n",
      "Precision (PPV):  0.9090909090909091\n",
      "Accuracy:  0.8709677419354839\n",
      "F1: 0.9090909090909091\n"
     ]
    }
   ],
   "source": [
    "# calculate some of these metrics using sklearn and the test set samples\n",
    "print(\"Sensitivity/Recall (TPR): \",metrics.recall_score(y_test,y_test_pred))\n",
    "print(\"Precision (PPV): \", metrics.precision_score(y_test,y_test_pred))\n",
    "print(\"Accuracy: \", metrics.accuracy_score(y_test,y_test_pred))\n",
    "print(\"F1:\", metrics.f1_score(y_test,y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can compute the full classification report, which will give us precision/recall per-feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Report:\n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.78      0.78      0.78        27\n",
      "           1       0.91      0.91      0.91        66\n",
      "\n",
      "   micro avg       0.87      0.87      0.87        93\n",
      "   macro avg       0.84      0.84      0.84        93\n",
      "weighted avg       0.87      0.87      0.87        93\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Classification Report:\\n\", metrics.classification_report(y_test,y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curves and AUC\n",
    "\n",
    "So far, we've working with the thresholded decisions a given classifier/model makes when it outputs a prediction (all we've been looking at is the predicted category of trained models). \n",
    "\n",
    "However, many classifiers (including both **logistic regression** and **random forest** models) can output a \"confidence\" associated with their prediction (this is called a **prediction probability**). \n",
    "\n",
    "Let's take a look at the prediction probabilites and the predictions of our original logistic regression classifier on our single test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic Regression predicted probabilities for first five samples in test set:\n",
      " [[0.018 0.982]\n",
      " [0.111 0.889]\n",
      " [0.011 0.989]\n",
      " [0.094 0.906]\n",
      " [0.01  0.99 ]]\n",
      "Logistic Regression predictions for first five samples in test set:\n",
      " [1 1 1 1 1]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class_0</th>\n",
       "      <th>class_1</th>\n",
       "      <th>predicted</th>\n",
       "      <th>actual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.018</td>\n",
       "      <td>0.982</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.111</td>\n",
       "      <td>0.889</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.011</td>\n",
       "      <td>0.989</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.094</td>\n",
       "      <td>0.906</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.010</td>\n",
       "      <td>0.990</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   class_0  class_1  predicted  actual\n",
       "0    0.018    0.982        1.0     1.0\n",
       "1    0.111    0.889        1.0     0.0\n",
       "2    0.011    0.989        1.0     1.0\n",
       "3    0.094    0.906        1.0     1.0\n",
       "4    0.010    0.990        1.0     1.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#lr probabilities per category for first five samples\n",
    "predicted_probs_lr = lr.predict_proba(X_test).round(3)\n",
    "predictions_lr = lr.predict(X_test)\n",
    "\n",
    "print(\"Logistic Regression predicted probabilities for first five samples in test set:\\n\",predicted_probs_lr[:5])\n",
    "print(\"Logistic Regression predictions for first five samples in test set:\\n\",predictions_lr[:5])\n",
    "y_test_lr_df = pd.DataFrame(\n",
    "    np.concatenate((\n",
    "        predicted_probs_lr,predictions_lr.reshape((predictions_lr.shape[0],-1)),\n",
    "        y_test.values.reshape((y_test.shape[0],-1))),axis=1\n",
    "    ),\n",
    "    columns = [\"class_0\",\"class_1\",\"predicted\",\"actual\"])\n",
    "\n",
    "y_test_lr_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets generate the same table of predicted probabilities, predictions, and actual values, for a trained random forest classifier that contains 100 trees:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class_0</th>\n",
       "      <th>class_1</th>\n",
       "      <th>predicted</th>\n",
       "      <th>actual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.08</td>\n",
       "      <td>0.92</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.40</td>\n",
       "      <td>0.60</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.04</td>\n",
       "      <td>0.96</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.05</td>\n",
       "      <td>0.95</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.22</td>\n",
       "      <td>0.78</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   class_0  class_1  predicted  actual\n",
       "0     0.08     0.92        1.0     1.0\n",
       "1     0.40     0.60        1.0     0.0\n",
       "2     0.04     0.96        1.0     1.0\n",
       "3     0.05     0.95        1.0     1.0\n",
       "4     0.22     0.78        1.0     1.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(n_estimators=100)\n",
    "rf.fit(X_train,y_train)\n",
    "\n",
    "predicted_probs_rf = rf.predict_proba(X_test)\n",
    "predictions_rf = rf.predict(X_test)\n",
    "\n",
    "y_test_rf_df = pd.DataFrame(\n",
    "    np.concatenate((\n",
    "        predicted_probs_rf,predictions_rf.reshape((predictions_rf.shape[0],-1)),\n",
    "        y_test.values.reshape((y_test.shape[0],-1))),axis=1\n",
    "    ),\n",
    "    columns = [\"class_0\",\"class_1\",\"predicted\",\"actual\"])\n",
    "\n",
    "y_test_rf_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In both of these cases, the prediction threshold (for either class) is set to 0.5, so the class with the probability that is over 0.5 is the predicted class.\n",
    "\n",
    "**However, 0.5 doesn't necessarily have to be the threshold we use for our classifier's decision threshold!**\n",
    "\n",
    "For example, we might want to be VERY CERTAIN that the classifier thinks something is class 1 (for example, a likelihood of cancer presence given some test results). \n",
    "\n",
    "In that case, we would require that the class 1 probability is > 0.8 (for example). As a result, the ratio of TP/TN/FN/FP would change, yielding a new confusion matrix, with a new set of precision/recall/accuracy estimates.\n",
    "\n",
    "**In general, the default probability threshold for a given classifier maximizes accuracy, but not the other metrics like precision, recall, etc.**\n",
    "\n",
    "But, what if we varied the prediction thresholds to obtain different TP/TN/FP/FN values (different confusion matrices) across all possible threshold values for our classifier?\n",
    "\n",
    "This is exactly what an [ROC (Receiver Operating Characteristic) curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) does.\n",
    "\n",
    "**An ROC Curve is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is systematically varied.**\n",
    "\n",
    "Let's generate the ROC curves for both of our models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'True Positive Rate (Sensitivity)')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcHVWZ//HPNxtNFpYsSEgInYSACZtADME1iEvCODDiFgQ0ikZggJ+CDIz4c2FGRUZGRUCMyDqyKmpEEJFJxGESkrAEQphISALphJHQhADZE575o6rN5aa7urrtuvd29/f9evWrb9U9derp00k/95yqOkcRgZmZWUt6VDsAMzOrbU4UZmaWyYnCzMwyOVGYmVkmJwozM8vkRGFmZpmcKMzMLJMThZmZZXKiMDOzTL2qHUBbDR48OOrr66sdhplZp/Lwww+/GBFD2nNsp0sU9fX1LFiwoNphmJl1KpKebe+xHnoyM7NMThRmZpbJicLMzDI5UZiZWSYnCjMzy1RYopB0raQXJC1q4X1JulzSUkmPSzqiqFjMzKz9iuxRXA9Mznh/CjAm/ZoO/KjAWMzMrJ0Ke44iIh6QVJ9R5ATgxkjWYp0raQ9JQyPi+aJisirbvg1e31rtKMxqSkSwedvr1Q4jUzUfuBsGrCzZbkj3OVF0RVs3wfcPhvVrqh2JWU0RUFftIFpRzUShZvZFswWl6STDU4wYMaLImKwoWzckSeKAKTDiqGpHY1Yz7nxkFate3sj4+oEFn+lf231kNRNFA7BvyfZwYHVzBSNiBjADYPz48c0mE+skRh8DR32+2lGY1YzfL3+Y5dvXc/Yn31XsiT7VORPFTOAsSbcCRwHrfH3CrDLOu30hf3raw4C14OWNWxk1uF+1w8hUWKKQdAswCRgsqQH4GtAbICKuBu4GjgOWAhuATxcVi5m90dxljfTbpRcTRxU93GF5vG304GqHkKnIu55OauX9AP6xqPObWbYjRuzJt088tNphWCfQ6aYZt4LM+hYsf6C4+rf7ttgsEcFzL21gzjONzF3WyKqXNxZ6vjWvbi60futanCgs8dgtsH0zDDmwmPp79obRx0L9O4upvxNa+dIG5ixLEsPcZxpZvW4TAIP778KYvfqj5u4L7CDj6/dkysF7F3cC61KcKGyH0cfCh/yAfFGeX7eROc80Jl/LGmlYm/QaBvbrw8RRAzlj1CCOHj2I0UP6oyKzhFkbOVF0Qa9s2sp3713Chi3bcx/z5Q1beObZl7jtjoUFRtY9bd3+OgtXvsyKxg0A7NG3N0eNHMhn3zGSo0cPZsxe/enRw4nBapcTRRf02HMvc+OcZxncvw+79OqZ65jNW7ez5pXNzNnUWHB03Y8Eb957N049up6JowYydu/dnBisU3Gi6MJ+fOqRHLlfztsfv7crQ+uHctyH3lNsUGbW6Xg9CjMzy+REYWZmmZwozMwskxOFpTzXopk1z4miu1vxIFz/QVi3Enbdo9rRmFkN8l1P3dVzc9NpO/4I/d8Ek78DR06rdlRmVoOcKLqblfNh9rfgmf+EfkPgA9+C8Z+B3rtWOzIzq1FOFN3Fqodh1rdh6X3QdxC872J462ehT23Pg29m1edE0dWtfgxmfxv+/DvYdU849mswYTrs0r/akZlZJ+FE0QlEBOs25p+m+7XN2xinFYy+/zp47j6o2x3e8xWY8Hmo263ASM2sK3Ki6AT+5a6nuPbB5bnLj9MK7upzEfG/A2DSl2Hi6UmyMDNrByeKTmD1yxsZMmAXzpw0Olf5fRpfpscjwfapN8Mor/9gZn+bzEQh6a3AKcA7gaHARmAR8Fvg5oh4tfAIDYCBffvw6bePzFd46TJ4BHr26lNsUGbWLbSYKCTdBTQCvwYuA14A6oADgGOA30q6NCLuqkSgZmZWHVk9itMi4i9l+zYB89Kv70jaq7DIzMysJrQ4hUdTkpB0uqRmr4RGxAtFBWZmZrUhz8XseuARSQ8B10bEH4oNqWvb/npwxn88zPPrNuU+ZkXjevbZ3U9Om1l1tJooIuJCSV8GpgCnS/oRcAtJ0lhRcHxdzisbt/L7xX9hzF792Xdg31zHDBmwC5MOHFJwZGZmzct1e2xEvC5pBbACOITkDqhfS7o7Iv65uPC6rpOPGsG0vHcxmZlVUauJQtKZwDTgFeCnwEURsVlSD2Ap4ERhZtaF5elRDAemRsSy0p1pL+P4YsIyM7NakWfhon3Kk4Sk6wEiYlERQZmZWe3I06M4tHQjHXJ6azHhdD7Pr9vILQ89x/bIt5Toxi2vFxyRmVnHynoy+wLgQmCApJeadpMsrvzTCsTWKfzy0VVc/p9L6dlDKOcxdb17MHKIp/k2s84hq0dxKcnUHd8mSRgARMT2ooPqTJo6Ek9dPJk+vbwEuZl1PVmJYv+IeFrSTcBBTTul5HNzRDxecGxV8UTDOp56/pXc5R9veLnAaFJrn4XlD+Qv/8JTxcViZt1OVqK4EDgNuLKZ9wJ4V2uVS5oM/ADoCVwTEZeUvT8CuAHYIy1zYUTcnS/0Ypxz66Msf3F9m47Zs29vevbIO/DUDvd/Axb9om3HqAf0G1xMPGbWrbSYKCLitPR7uxY0kNSTJMm8D2gA5kuaGRGLS4p9Bbg9In4kaRxwN8mUIVWzZdvrTDl4b77ywXG5j9l914ITxfYtMGh/OPVX+Y/p0w/6DiwuJjPrNvI8cPcIyZQdt0fEs22oewKwtOnWWkm3AicApYkigKa1OXcHVreh/sL07dOLYXvU2NxKPfvAHvtWOwoz64byXH39KNAbmClpjqQvSBqW47hhwMqS7YZ0X6mvA6dIaiDpTZydo14zM6ugVhNFRDwTEd+KiMOAzwBHAnl6Fs2NxZQ/bHAScH1EDAeOA25Kn9N4Y0XSdEkLJC1Ys2ZNjlObmVlHyXU/p6Thks4FrgfGARflOKwBKB0rGc7OQ0unAbcDRMQckhX0droCGxEzImJ8RIwfMsSzqJqZVVKriULSgyTDQv2AUyPiyIj4To665wNjJI2U1AeYCswsK/MccGx6nrEkicJdBjOzGpJnCo/Pt2dOp4jYJuks4F6SW1+vjYgnJV0MLIiImcB5wE8kfZFkWGpaRM65MMzMrCKypvA4KSJuAd4j6T3l70fE5a1Vnj4TcXfZvq+WvF4MvL1NEZuZWUVl9Sj2TL83d1HAn/rba9sWuHICvLIq/zHbt8KbDmq9nJlZAbIeuLsqffnbiJhb+p6kiYVG1ZVteQ3WLodRk2Cfw/Mft987iorIzCxTnmsUVwFHlO27kuQ2WWuvA4+Doz5f7SjMzFqVdY1iAnA0METSOSVv7UbyAJ6ZmXUDWT2KfiTPNPTijdcpXiV5WtvMzLqBrGsUs4BZkq4rXwrVzMy6j6yhp8si4jzgMkk73eUUEScWGlkHWLt+C1+d+SQbt2zLfcyLr20uMCIzs84na+jptvT7FZUIpAhPrFrHbxauZuTgfvTt0zPXMfvv1Z93H+hpQszMmmQNPc1Lv9/ftE/S7sCwsjUlat53P3ooR+7ntRnMzNojz3oU9wMfIpmGYyHwkqT7IuL8ooPrFNY+C/NmQLyer/zWjcXGY2bWwfI8RzEwIl6RdBpwQ0T8f0mPA04UAIt+DnOugD4DQDlXudt1IAw5sNi4zMw6SJ5E0UvSEJJbYr/aWuFup2kOw39aBr36VDcWM7MC5FmP4pvAH4HnImKepFHA8mLDMjOzWtFqjyIibgVuLdleRrL2tZmZdQN5LmYPJlkCtb60fERMLy4sMzOrFXmuUfwamAv8F7C92HBa9+qmbcxe8kKusotWrSs4GjOzri9PouiXPqFdE1Y0rmfadfPbdMyAOs9haGbWXnkSxT2S3h8Rvy88mhz69OzBnWe+LXf5Abv0YsybBhQYkZlZ15YnUZwOXCBpA7AFEBARUZVHnXtIHDFiz9YLmplZh8iTKAYXHoWZmdWsVp+jiIjtJA/bXZC+Hgq8pejAzMysNrSaKCRdARwDnJru2gBcXWRQZmZWO/IMPb0tIo6Q9ChARLwkyXNVmJl1E3mm8NgqqQcQAJIGATmnSjUzs84uT6K4EvgFMETSN0gevPtOoVGZmVnNyDPX042SHgbem+76aEQsKjasDvLyc3Dzx2HLa8WdY5Of/jazri1rzew6YGtEbI+IJyVtBqYAo4DOkShefBpeWAz7vxf6Fbi86cDRnmLczLqsrB7FvcDngD9LGg3MI1lH+8OSjoqIiyoRYId41z/BiKOqHYWZWaeUdY1iYET8OX39KeDWiDgD+ABwfOGRmZlZTchKFFHy+j3AfQARsRnf9WRm1m1kDT09KekSYBVwAPB7AEm7k8z3ZGZm3UBWj+KzwGvAm4HJEbE+3X8w8O9FB2ZmZrWhxR5Fmhj+tZn9DwIPFhmUmZnVjhZ7FJJ+JWmKpJ2SiaT9JH1V0meyKpc0WdISSUslXdhCmY9JWizpSUk3t/1HMDOzImVdo/hH4DzgSkl/AdYAdSTPUTwHXBkRv2jpYEk9SZ7qfh/QAMyXNDMiFpeUGQP8M/D2iFgraa+/9Qd6g22bO7Q6M7PuKGvoaRVwLnCupP1JphffCCyJiFdz1D0BWBoRywAk3QqcACwuKfM5koSzNj1nvsWwW/PaC/DgD2D+T6FHb+jnJTXMzNorz+yxRMRSYGkb6x4GrCzZbgDKn3o7AEDSg0BP4OsR8bs2nmeH9S+mCeIa2LYJDp0K7z4fBo5qd5VmZt1drkTRTs3dQhtl272AMcAkYDjwJ0kHR8TLb6hImg5MB+g7dPTOtW54Cf77cnhoBmzbCId8NHkae/D+f/tPYWbWzRWZKBqAfUu2hwOrmykzNyK2AsslLSFJHPNLC0XEDGAGwO77vnlHstnwEsy5Eh66Grash4M/DO++AIYc0PE/jZlZN5UrUaQLFY1Ih6Dymg+MkTSS5KG9qcAnysr8CjgJuF7SYJKhqGWt1rzxZZh7Fcz9EWx+BQ76UJIg9hrbhvDMzCyPVhOFpL8jecCuDzBS0luAr0XEh7KOi4htks4imVywJ3BtOgvtxcCCiJiZvvd+SYuB7cD5EdGYVe/AeAm+fyhsXgdjj4dJF8KbDsrzs5qZWTsoovyyQVmBZC2KY4FZEXF4uu+JiDikAvHtZPw+PWPBZR9PEsTQQ6sRgplZpyPp4YgY355j8ww9bY2Il6U3XJvOzi4F2qI+cJKfyzMzq5Q8ieIpSR8DeqTXG/4fMLfYsFoWno/QzKyi8qyZfRZwJMnU4ncCm0iShZmZdQN5ehQfiIgLgAuadkg6kSRpmJlZF5enR/GVZvZ1nmVQzczsb9Jij0LSB4DJwDBJpetP7IZXuDMz6zayhp5eABaRXJN4smT/q0CzU4abmVnXkzV77KPAo5J+FhGbKhiTmZnVkDwXs4dJ+iYwjmQ9CgAiwhMqmZl1A3kuZl8PXEcyG+wU4Hbg1gJjMjOzGpInUfSNiHsBIuKZiPgKcEyxYZmZWa3IM/S0Wcn8Hc9IOp1kJtiOXbLUzMxqVp5E8UWgP3AO8E1gd+AzRQZlZma1o9VEEREPpS9fBU4FkDS8yKDMzKx2ZF6jkPRWSf+QLiqEpIMk3UgVJwU0M7PKajFRSPo28DPgZOB3ki4CZgELSVaiMzOzbiBr6OkE4LCI2ChpIMl614dFxJLKhGZmZrUga+hpU0RsBIiIl4D/cZIwM+t+snoUoyQ1TSUuoL5km4g4sdDIzMysJmQlig+XbV9RZCBmZlabsiYFvL+SgZiZWW3KM4WHmZl1Y04UZmaWKXeikLRLkYGYmVltajVRSJog6Qng6XT7MEk/LDwyMzOrCXl6FJcDHwQaASJiIZ5m3Mys28iTKHpExLNl+7YXEYyZmdWePNOMr5Q0AQhJPYGzgT8XG5aZmdWKPD2KM4BzgRHAX4CJ6T4zM+sG8vQotkXE1MIjMTOzmpSnRzFf0t2SPiVpQOERmZlZTWk1UUTEaOBfgSOBJyT9SpJ7GGZm3USuB+4i4r8j4hzgCOAVkgWNWiVpsqQlkpZKujCj3EckhaTxuaI2M7OKyfPAXX9JJ0v6DTAPWAO8LcdxPYErgSnAOOAkSeOaKTcAOAd4qPw9MzOrvjw9ikUkdzpdGhH7R8R5EZHnj/oEYGlELIuILcCtJKvmlfsX4FJgU96gzcyscvLc9TQqIl5vR93DgJUl2w3AUaUFJB0O7BsRd0n6UjvOYWZmBWsxUUi6LCLOA34hKcrfz7HCnZrZ99d6JPUAvgdMay1ISdOB6QAHD61rrbiZmXWgrB7Fben39q5s1wDsW7I9HFhdsj0AOBiYLQlgb2CmpOMjYkFpRRExA5gBcMiwvjslLTMzK07WCnfz0pdjI+INyULSWUBrK+DNB8ZIGgmsAqYCnyipfx0wuKTO2cCXypOEmZlVV56L2Z9pZt9prR0UEduAs4B7gaeA2yPiSUkXSzq+bWGamVm1KKL5kRxJHyfpBUwCZpW8NQDoFRFVmWr8kGF944lVG6pxajOzTkvSwxHRrmfVsq5RzCNZg2I4yfMQTV4FHm3PyczMrPPJukaxHFgO/KFy4ZiZWa3Juj32jxHxbklrKbmtleS214iIgYVHZ2ZmVZc19NR0DWJwRhkzM+viWrzrqeRp7H2BnhGxHTga+DzQrwKxmZlZDchze+yvSJZBHQ3cCIwFbi40KjMzqxl5EsXrEbEVOBH4fkScTTKPk5mZdQN5EsU2SR8FTgXuSvf1Li4kMzOrJXmfzD6GZJrxZemUHLcUG5aZmdWKFp/MfkMhqRewf7q5NJ2eoyr8ZLaZWdsV9WR2U+XvBG4imdhPwN6STo2IB9tzQjMz61zyLFz0PeC4iFgMIGksSeLw+tZmZt1AnmsUfZqSBEBEPAX0KS4kMzOrJXl6FI9I+jFJLwLgZDwpoJlZt5EnUZwOnAP8E8k1igeAHxYZlJmZ1Y7MRCHpEGA08MuIuLQyIZmZWS1p8RqFpC+TTN9xMnCfpOZWujMzsy4uq0dxMnBoRKyXNAS4G7i2MmGZmVmtyLrraXNErAeIiDWtlDUzsy4qq0cxStKd6WsBo0u2iYgTC43MzMxqQlai+HDZ9hVFBmJmZrUpa83s+ysZiJmZ1SZfdzAzs0xOFGZmlil3opC0S5GBmJlZbWo1UUiaIOkJ4Ol0+zBJnsLDzKybyNOjuBz4INAIEBELSVa8MzOzbiBPougREc+W7dteRDBmZlZ78sweu1LSBCAk9QTOBv5cbFhmZlYr8vQozgDOBUYAfwEmpvvMzKwbaLVHEREvAFMrEIuZmdWgVhOFpJ8AUb4/IqYXEpGZmdWUPNco/lDyug74ELCymHDMzKzW5Bl6uq10W9JNwH15Kpc0GfgB0BO4JiIuKXv/XOCzwDZgDfCZZu6wMjOzKmrPFB4jgf1aK5TeIXUlMAUYB5wkaVxZsUeB8RFxKPBzwMutmpnVmDzXKNay4xpFD+Al4MIcdU8AlkbEsrSeW4ETgMVNBSJiVkn5ucAp+cI2M7NKyUwUkgQcBqxKd70eETtd2G7BMN54LaMBOCqj/GnAPS3EMR2YDnDw0Lqcpzczs46QOfSUJoVfRsT29CtvkoBkVbydqmy2oHQKMB74txbimBER4yNiPGquWjMzK0qeaxTzJB3RjrobgH1LtocDq8sLSXovcBFwfERsbsd5zMysQC0OPUnqFRHbgHcAn5P0DLCepKcQEdFa8pgPjJE0kmToairwibJzHA78GJicPthnZmY1JusaxTzgCOAf2lNxRGyTdBZwL8ntsddGxJOSLgYWRMRMkqGm/sAdyeUQnouI49tzPjOzomzdupWGhgY2bdpU7VBaVVdXx/Dhw+ndu3eH1amWLjtIejQiDu+wM3WQQ4b1jSdWbah2GGbWjSxfvpwBAwYwaNAgVMPXSSOCxsZGXn31VUaOHPmG9yQ9HBHj21NvVo9iSPpAXEsB/Xt7Tmhm1tls2rSJ+vr6mk4SAJIYNGgQa9as6dB6sxJFT5JhodpuGTOzCqj1JNGkiDizEsXzEXFxh5/RzMzarH///rz22mtVOXfW7bGdI32amXVT27dXZrHRrERxbEUiMDOz3GbPns0xxxzDJz7xCQ455JCKnLPFoaeIeKkiEZiZdSLf+M2TLF79SofWOW6f3fja3x+Uu/y8efNYtGjRTnc2FaU9s8eamVkVTZgwoWJJAvItXGRmZqm2fPIvSr9+/Sp6PvcozMwskxOFmZll8tCTmVkn0PQMxaRJk5g0aVJFz+0ehZmZZXKiMDOzTE4UZmaWyYnCzMwyOVGYmVkmJwozM8vkRGFm1gndcccdjB07lmOOOabwczlRmJl1MhHBT37yE6666ipmzZpV+PmcKMzMOoEVK1YwduxYzjzzTHr06MF9993H6aefzvnnn1/4uf1ktplZW9xzIfzvEx1b596HwJRLWi22ZMkSrrvuOq666iomTZrEd7/7XcaPH9+xsTTDPQozs05iv/32Y+LEiRU/r3sUZmZtkeOTf1EqPb14E/cozMwskxOFmZll8tCTmVknUF9fz6JFi/66PXv27Iqd2z0KMzPL5ERhZmaZnCjMzCyTE4WZWQ4RUe0QcikiTicKM7NW1NXV0djYWPPJIiJobGykrq6uQ+v1XU9mZq0YPnw4DQ0NrFmzptqhtKquro7hw4d3aJ2FJgpJk4EfAD2BayLikrL3dwFuBI4EGoGPR8SKImMyM2ur3r17M3LkyGqHUTWFDT1J6glcCUwBxgEnSRpXVuw0YG1E7A98D/hOUfGYmVn7FHmNYgKwNCKWRcQW4FbghLIyJwA3pK9/DhwrSQXGZGZmbVRkohgGrCzZbkj3NVsmIrYB64BBBcZkZmZtVOQ1iuZ6BuW3DOQpg6TpwPR0c7OkReVluqnBwIvVDqJGuC12cFvs4LbY4cD2HlhkomgA9i3ZHg6sbqFMg6RewO7AS+UVRcQMYAaApAURUfxKHZ2A22IHt8UObosd3BY7SFrQ3mOLHHqaD4yRNFJSH2AqMLOszEzgU+nrjwD/GbV+o7KZWTdTWI8iIrZJOgu4l+T22Gsj4klJFwMLImIm8FPgJklLSXoSU4uKx8zM2qfQ5ygi4m7g7rJ9Xy15vQn4aBurndEBoXUVbosd3BY7uC12cFvs0O62kEd6zMwsi+d6MjOzTDWbKCRNlrRE0lJJFzbz/i6Sbkvff0hSfeWjrIwcbXGupMWSHpd0v6T9qhFnJbTWFiXlPiIpJHXZO17ytIWkj6X/Np6UdHOlY6yUHP9HRkiaJenR9P/JcdWIs2iSrpX0QkuPEChxedpOj0s6IlfFEVFzXyQXv58BRgF9gIXAuLIyZwJXp6+nArdVO+4qtsUxQN/09RnduS3ScgOAB4C5wPhqx13FfxdjgEeBPdPtvaoddxXbYgZwRvp6HLCi2nEX1BbvAo4AFrXw/nHAPSTPsE0EHspTb632KDz9xw6ttkVEzIqIDenmXJJnVrqiPP8uAP4FuBTYVMngKixPW3wOuDIi1gJExAsVjrFS8rRFALulr3dn52e6uoSIeIBmnkUrcQJwYyTmAntIGtpavbWaKDz9xw552qLUaSSfGLqiVttC0uHAvhFxVyUDq4I8/y4OAA6Q9KCkuelszl1Rnrb4OnCKpAaSOzHPrkxoNaetf0+A2l2PosOm/+gCcv+ckk4BxgPvLjSi6slsC0k9SGYhnlapgKooz7+LXiTDT5NIepl/knRwRLxccGyVlqctTgKuj4jLJB1N8vzWwRHxevHh1ZR2/d2s1R5FW6b/IGv6jy4gT1sg6b3ARcDxEbG5QrFVWmttMQA4GJgtaQXJGOzMLnpBO+//kV9HxNaIWA4sIUkcXU2etjgNuB0gIuYAdSTzQHU3uf6elKvVROHpP3ZotS3S4ZYfkySJrjoODa20RUSsi4jBEVEfEfUk12uOj4h2z3FTw/L8H/kVyY0OSBpMMhS1rKJRVkaetngOOBZA0liSRFH7y9V1vJnAJ9O7nyYC6yLi+dYOqsmhp/D0H3+Vsy3+DegP3JFez38uIo6vWtAFydkW3ULOtrgXeL+kxcB24PyIaKxe1MXI2RbnAT+R9EWSoZZpXfGDpaRbSIYaB6fXY74G9AaIiKtJrs8cBywFNgCfzlVvF2wrMzPrQLU69GRmZjXCicLMzDI5UZiZWSYnCjMzy+REYWZmmZwouilJ2yU9VvJVn1G2vqXZKNt4ztnpDJ8L02kl2rzYu6TTJX0yfT1N0j4l710jaVwHxzlf0ltyHPMFSX3bca7vS3pX+vqsdFbPSJ97aGtdB6axPybpKUkdumiPpOObZmaVNETJrM2PSnqnpLsl7ZFxbIu/t4xj/iBpz477Cazdqj3bob+q8wW81oay9bQwG2UbzzmbdDZXYDows6Pq6+C2KY3z08B9OY5ZAQxu43kGAnNLtg9P27rNdaXH3wucULJ9SIH/fqYCNxT5eyN5oPaion4Gf+X/co/C/irtOfxJ0iPp19uaKXOQpHnpp9bHJY1J959Ssv/Hknq2croHgP3TY49NP5k+oWQ+/V3S/Zdoxzob3033fV3SlyR9hGReq5+l59w1/TQ9XtIZki4tiXmapB+2M845lEyaJulHkhYoWd/hG+m+c4B9gFmSZqX73i9pTtqOd0jq30zdHwF+17QREY9GxIpW4skylGSKhqb6nkhjmSbp15J+l/aUvlby8zTbHkrWd3gk7VXdX1LPFWkP61LguJK2X9HUC5L0yfR3tlDSTem+ln5vfyfplyXxvE/SnenmTJI5mqzaqp2p/FWdL5IndR9Lv36Z7usL1KWvx5A81QolPQrgh8DJ6es+wK7AWOA3QO90/1XAJ5s552x2fFI/H7iNZCqFlcAB6f4bgS+QfNpewo6HQvdIv38d+FJ5faXbwBCSaaeb9t8DvKOdcX4B+FbJewPT7z3Tcoem2ytIewEkcwg9APRLty8AvtrMeW4A/r6Z/X+tq42/00+TzKJ8D/DFkjabBjxPMrvyrsCitJ2abY+0/VYCI8t+5mnAFeWvS2MGDkp/b4PLjm3290YySd3/AEPS7ZtL2wR4GhhU7f8v3f2rJqfwsIrYGBHlY++9gaZPjNtJ5gYqNwe4SNJw4M6IeFrSscCRwHwlU4jsCrQ059QDwltQAAADYElEQVTPJG0k+cNyNnAgsDwi/py+fwPwj8AVJOtJXCPpt0DuacMjYo2kZUrmsnk6PceDab1tibMfSUIoXQXsY5Kmk0x/M5RkEZzHy46dmO5/MD1PH5J2KzeUDpxvKCKuk3QvMJlk3YHPSzosffu+SKfvSD+xvwPYRvPtMRF4IJKJBImItky2+R7g5xHxYp5jIyLSXscpkq4DjiZJVk1eIOmtdbmpRzoTJwor9UXgL8BhJDc67LTwT0TcLOkh4O+AeyV9luRT4Q0R8c85znFylEzSJ6nZNUQimb9nAslEblOBs0j+COV1G/Axkk+rv0z/ILUpTpKV0i4BrgROlDQS+BLw1ohYK+l6kh5ROZH8YW5t2GRjC8e3KP1jejiwOiJ2Ws4zIlYD1wLXKrkB4eCmt8qL0sLvTdLxzZTPHWI7jr2OpGezCbgjkvVlmtSRtJNVka9RWKndgecjmaP/VJJP028gaRSwLCIuJxlDPhS4H/iIpL3SMgOVf93u/wHqJe2fbp8K/DEd0989Iu4mGf5p7s6jV0mmFm/OncA/kIxx35bua1OcEbEV+AowUcmMo7sB64F1kt4ETGkhlrnA25t+Jkl9JTXXO3uK9DpNXhHx6Yh4S3NJIr2u0Dt9vTfJUNOq9O33pT/vriTt8iAtt8cc4N1pYkTSwDaEeD9Jr2tQxrFv+L2lyW01SVtfX/LzCNibpPdpVeREYaWuAj4laS7JsNP6Zsp8HFgk6THgzSTLKi4m+U/+e0mPA/eRDKu0KiI2kYyt3yHpCeB14GqSPyR3pfX9kaS3U+564OqmC6pl9a4FFgP7RcS8dF+b44yIjcBlJOPrC0nWoH6S5FP7gyVFZwD3SJoVEWtIxvBvSc8zl6Styv2WZKZPILkormTGz+HA45KuyYqtGe8n+d0sJLkD6vyI+N/0vf8CbiK5JvWLiFjQUnuk8U8H7kzruq38RC2JiCeBb5Ik+4XAvzdT7Hp2/r39DFiZxtTkSJK7wraVV2CV5dljzapI0n8BH4wCV52TNI3k4vFZRZ3jbyXpCuDRiPhpyb4fkNxCfX/1IjNwj8Ks2s4DRlQ7iGqS9DDJEOZ/lL21yEmiNrhHYWZmmdyjMDOzTE4UZmaWyYnCzMwyOVGYmVkmJwozM8vkRGFmZpn+D6WjWf/cjM1uAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a19477400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#generate lr model false positive and true positive rates\n",
    "fpr_lr, tpr_lr, thresholds_lr = metrics.roc_curve(y_test, predicted_probs_lr[:,1])\n",
    "\n",
    "#generate same for random forest model\n",
    "fpr_rf, tpr_rf, thresholds_rf = metrics.roc_curve(y_test, predicted_probs_rf[:,1])\n",
    "\n",
    "# plot LR and RF model ROC curves\n",
    "sns.mpl.pyplot.plot(fpr_lr, tpr_lr,label=\"lr\")\n",
    "sns.mpl.pyplot.plot(fpr_rf, tpr_rf,label=\"rf\")\n",
    "sns.mpl.pyplot.xlim([0, 1])\n",
    "sns.mpl.pyplot.ylim([0, 1.05])\n",
    "sns.mpl.pyplot.legend(loc=\"lower right\")\n",
    "sns.mpl.pyplot.xlabel('False Positive Rate (1 - Specificity)')\n",
    "sns.mpl.pyplot.ylabel('True Positive Rate (Sensitivity)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ROC curve allows you to balance how many FN's (misses) and FP's (false alarms) you're willing to have with respect to one of your classes.\n",
    "\n",
    "There's a metric that allows you to quantify the overall performance of your binary classifier, regardless of the threshold you choose. This metric is called the **AUC (area under the curve)**, and is a systematic way to compare two classifiers relative to each other, across all decision thresholds. If model A has a higher AUC than model B then this means that on average, model A is a better classifier across all decision thresholds (all probabilities). \n",
    "\n",
    "Besides allowing you to calculate AUC, seeing the ROC curve can help you to choose a threshold that **balances sensitivity and specificity** in a way that makes sense for the particular context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR model AUC:  0.9357463524130191\n",
      "RF model AUC:  0.9281705948372614\n"
     ]
    }
   ],
   "source": [
    "# calculate AUC for lr and rf\n",
    "print(\"LR model AUC: \",metrics.roc_auc_score(y_test, predicted_probs_lr[:,1]))\n",
    "print(\"RF model AUC: \",metrics.roc_auc_score(y_test, predicted_probs_rf[:,1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make this more explicit. Let's say instead of using 0.5 as the probability threshold for class 1, we want the probability threshold where the false positive rate is no higher than 12% (because additional false positives cost too much money, or time, etc.). \n",
    "\n",
    "Let's take a look at a graph of the ROC curve and the class 1 thresholds as a function of FPR:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'True Positive Rate (Sensitivity) or Class 1 Threshold Probability')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAFJCAYAAACFAmoKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VPW9//HXJ2EJ+xZUVkFFEVFQFhfUq1Jbra221AVbtai91lrstVp/2ttbW7ldrL2999aqte6trUtttVK11wXFHRARZbEqIMqigggIyJbk8/vjewYmYTJzMsmZmSTv5+Mxj8xZ5pxPTiCffHdzd0REROpTVuwARESktClRiIhIVkoUIiKSlRKFiIhkpUQhIiJZKVGIiEhWShQiIpJVzkRhZheaWbdCBCMiIqUnToliEDDHzO42s88kHI+IiJQYizMy28zKgBOBc4ERwD3A7e6+NNHoRESk6GK1Ubh7DbA0etUAfYCHzOzniUUmIiIlIWeJwswuAiYBnwC3AQ+4+9aolLHI3fdKPEoRESmaNjHO6Q9MdPcl6TvdvcbMTk4mLBERKRVxqp761k0SZnYngLvPTyIoEREpHXESxUHpG1GV05hkwhERkVJTb6IwsyvMbC1wkJl9HL3WAh8BjxYsQhERKap6G7PNzIBy4OfAlan97l5dmNBERKQUZEsUQ9z9bTM7KNNxd3890chERKQkZEsUt7v7eWb2XIbD7u5HJxuaiIiUglgjs0VEpPWqdxxFrjES7j616cMREZFSk23A3WlZjjmgRCEi0gqo6klERLLKVvV0prvfY2bfyXTc3a9LLiwRESkV2aqeekRfexciEBERKU2qehIRkaziLIU6yMweNLMPotdfzWxQ8qGJiEgpiDMp4D2EHk4Do9ffo30iItIKxEkUZe5+h7tvi153xvyciIi0ANl6PXWN3j5lZt8D7iWMnziDUKoQEZFWINtcT8sIicEyHHZ3H5hkYCIiUhrU60lERLKKs2Y2ZjYUGAZUpPa5+91JBSUiIqUjZ4nCzP4D+CwwFHgM+BzwvLtPSD48EREptji9l84AjgXed/ezgRHELImIiEjzFydRbI6WP60ysy7AB8BeyYYlIiKlIk7J4FUz6w7cDswGPgHmJBqViIiUjAb1ejKzfYCu7q5EISLSSsTt9XQycCRhXMXzqEQhItJqxOn19BtC19h7o12nA/9094sTjk1EREpAnESxABju0YlmVg687u4HFCA+EREpsji9nt4C+qdt9wHmJxOOiIiUmmyTAj5IaJPoBrxhZjOiQ4cBLxQgNhERKQHZGrOvL1gUIiJSsmJ1jzWzSmB0tDnb3T9KNCoRESkZcZZC/QqhO+zZwDnAbDP7ctKBiYhIaYjT6+k14LPu/mG0vTvwuLuPKEB8IiJSZHGXQv0wbXt1zM+JiEgLEGdk9hNm9iiQWn9iImG6cRERaQXiVD0ZcBphCg8DngX+4loaT0SkVciaKKJR2I+6++cKF5KIiJSSrG0N0ToU28ysa4HiERGREhOnjWIj8JqZPQ5sSu1090sTi0pEREpGnETxZPQSEZFWKGuiMLMDgTXAAnd/uzAhiYhIKam3MdvM/h04nzAqewwwxd1vL2BsGVV2buuDhmqsn4hIQ7zyyisfuXvvfD6bLVEsAMa6+yYz603o/TSmEXE2idF7dvbZ724sdhgiIs2Kmb3i7qNzn7mrbL2etrr7JgB312hsEZFWKlsbxV5m9kD03oC907Zx9wmJRiYiIiUhW6L4Sp1trU8hItIK1Zso3H1aIQMREZHSlFi7g5ndbmarzCzj+toWXGdmi8zsdTM7JKlYREQkf0k2UN8JnJDl+InAkOh1AfDbBGMREZE8JZYo3P1Z4OMsp5wC/MGDGUB3M+uTVDwiIpKfetsozOxBoN6pZZug11M/YFna9vJo3/uNvK6ISLMx9bWV3D3z3WKHkVW2EsX1wA2EX+A1wF3Rqwp4swnubRn2ZUxMZnaBmc02s9lVVVVNcGsRkdLw6Ovv89qy9dQ4ib4aI2evJzP7kbsfndpvZn8DnmncbYGQgAakbfcHVtYTy83AzRBGZjfBvUVESsbAnh358zcPT/QedmH+n40ze+xuZjbI3ZdG2wOBvOYLqWMqMNnM7gUOBda7u6qdRArg/tnLmL10bbHDEGDeivV0bh/nV3HxxInuMuA5M0tVNw0BvpXrQ2Z2D3AMUGlmy4EfAW0B3P0m4FHg88Ai4FPg3IYGLyL5+d8n32bNpq1079Cu2KEIcPjevYodQlY5E4W7P2Jm+wLDol0L3X1zjM+dmeO4A9+OFaWINLmTDuzLr07XTMySW7ZeTyfXc6ifmeHuUxOKSaTVWvXJFl5asoYV63L+LdYoG7ZsT/T60rJkK1GcluWYE9oYRKQRVm/Yyowla5ixZA0vLVnDktWbcn+oiQzq1bFg95LmLVuvp7MLGYhIa/Dxpm07E8PiNby9Kqyt0rl9G8YM6sHEMQM4fK9KhuzeOfFYKtqWJ34PaRlytlGYWRfgh0Cqi+wzwE/cfUOSgUn+qmucZ99azafbqosdigDbq2uYu2wdM5as4Z8fhP82HduVM3pQTyYc0p/D9+7F8L5daVOuJV+kNMXp9XQ78BZwTrR9NnAHcGpSQUnjvPLuWs698+VihyFpKtqWMXrPnlz+ub4ctlcvDurfjbZKDNJMxEkUQ9w9vb3ih2Y2N6mApPG2bA8lif85YwQH9O1W5GikzGBgz060a6PEIM1TnESxxcwOd/eXAMzsMGBLsmFJUxjYsyP77t6l2GGISDMXJ1FcBNxlZu0J8zN9ys5qKBERaeHiDLibAxxgZj0Bc/c1yYclIiKlIk6vp3bAl4BBQBuzMOmru/8s0chERKQkxKl6epDQJvEKoP6WIiKtTJxEsae7D088EhERKUlx+uvNMLNhuU8TEZGWKNukgK8S5nRqC8w1s7eBrYSeT+7uhxQmRBERKaZsVU8aeS0iIvVXPbn7YndfTGjAXha97wN8DlhdoPhERKTI4jRm/w0YY2Z7A38AHgHuBr6QZGD18hpYuxTadIA27aFtByhvB1G33Zbor68s564Z78Y+/xOtNSAiTShOoqhx9+1mNgH4X3e/Lmq/KI6qrfDruqtyGbSp2Jk42lSEV9sK6FgJX7kFOvQoSrhN4YmFH/LWhxsYPahnrPO7dmjL/nt01fQdItIk4iSKKjM7jTBr7JeifW2TCymHHoPglClQtTkkjaotsH1L+Jp6pbY/WQGLnoDVb8HAQ4sWclMY0KMjfzhvbLHDEJFWKE6iOI8w39O17r7EzAYD9yQbVhYdesDBX4t37qJp8McJycYjItLCxZnrab6ZXQIMjLbfAX6adGAiIlIacg64M7OTgHnAE9H2SDN7MOnARESkNMSpepoCHAo8DeDuc81sn0SjauHumfUe76/bHPv8t1ZtoG2ZFr0RkeKIkyi2u/s6q9391BOKp8Vb/+l2vv/APKBhPXpPOrBPQhGJiGQXJ1G8YWanA2VRQ/a/ATOSDavlqvGQY3/8xWFMGje4yNGIiOQWpz5jMjAKqAEeIEw5fkmSQYmISOnIWqIws3Lgh+5+BXBFYUISEZFSkrVE4e7VgEZ5iYi0YnHaKOaY2QPA/cCm1E53n5pYVM3I9uoa5q1YT01NvPb9DVuqEo5IRKRpxUkUuxMSxOfT9jnQfBLFoiehrBwq94WKrk166btnvsePpi5o8Oc6to/z6EVEii/OyOyzCxFIIroPhPZd4dlrwwuga3/YbSj0jl677Q+994P2+U2gt3FrKCHcMWkMbcrj9XdtU1bG6EHNd5JCEWldciYKM6skzPc0KP18d78gubCaSOUQuGJpmJZ89T9h1Ruw+k1Y/QYsfT5MHJgSJZDqyqGsG3oG1b32jXWLVKIYt08l7dpoUJyItDxx6j8eIoybeJ6wiFHzUlYOvfYOr6En7dxfU505gSx6iiefn88VVfHzYHmZteTlMESklYuTKDq5+2X5XNzMTgB+DZQDt7r7NXWODwR+D3SPzrnS3R/N514NVk8CWfXjfejduQ0/OWZ47EsN6NmRtuUqTYhIyxQnUfzDzD7r7o835MLRGIwbgOOB5cDLZjbV3RemnfYfwJ/d/bdmNgx4lFDFVVRd2rfhrMP2LHYYIiIlod5EYWZrCb2bDLjCzD4FtkXb7u65llsbCyxy9yXR9e4FTgHSE4UDqW5I3YCV+XwTIiKSnGwligOB9xtx7X7AsrTt5YRZaNP9GHjczC4GOgGfyXQhM7sAuABg4MCBjQhJREQaKlvF+lR3r67vFePamZp3645KOxO40937E8Zp3GVmu8Tk7je7+2h3H927d+8YtxYRkaaSLVE0th/PcmBA2nZ/dq1aOh/4M4C7vwRUAJWNvK+IiDShbFVP/czsv+s76O6X5rj2y8CQaGryFcBE4Kt1znkPGA/caWb7ExLF6pxRi4hIwWRLFJuBhs9NEXH3KjObDDxG6Pp6u7svMLMpwOxorqjLgFvM7LuEaqlJ7q5FkURESki2RLHG3W9rzMWjMRGP1tl3Vdr7hcC4xtxDRESSla2NovmNwhYRkSZXb6Jw9zGFDEREREqT5p0QEZGslChERCSrvBKFmVU0dSAiIlKa8l1m7S2g5OfS2F5dw8Ovr2TT1vjt8sclGI+ISHOUbVLA79R3COicTDhNa/bStXz3vtca9Jnn2zsVbcsTikhEpPnJVqK4FvhvMneTbRZtG9urawC49ZzRHDSgW6zP9LqlA/36Ne262iIizVm2RDEH+Iu7z6l7wMwmJRZRAnp0astuXWI2q5iFJVKrt0N522QDExFpBrIlim9Q/7xLhyUQS2lo2wEWPAgLp0K3/tBjUO1Xz8Hha4ceRQ1TRKRQ6k0UdVaiq3tsRTLhlIAz74H3ZsDad8Ka2muXwpuPwqY6ObOi287kMfAIOOzCwscqIlIA+fZ6arlS62jXtXUDrH13Z/JYuzQkk3dfgoUPwejzoE27AgcrIpI8JYq42neBPYaHV7oXfg1PXAW3Hgd7HQt7HwsDDw9VWCIiLYASRWMdfDZUb4PF02HGb+HF66C8PQw8DPY6Jrz6jIAydbkVkeYpZ6Iws58DPwc+BR4BRgLfdfe7E46teejYE46+PLy2bYJ3X4Ql08Nr2tXh1aEHDD46ShzHhgZxEZFmIk6J4kR3/76ZfQlYBRwATAOKkijmrVjP4O8/Euvc1BJIZdbYVV1jatcJhhwfXgAbPoR3noUlT4fEsfChsL/HIDj1Duh3SGHiEhFphDiJInXO54F73P0jMyvaKnRtyoyLj90n9vmdK9owvF+8wXZNrsvucNBp4eUOaxaFHlRPXAXLX1aiEJFmIU6i+IeZzSeM0P62mVUCW5MNq35tysq49LP7Fev2+TODyiHQ8eyQKEREmomcU3G4++WEufJGuft2wlraE5IOTERESkPORGFmE4DN7l5lZlcCdwC9E49MRERKQpzJ/X7s7hvM7Ajgi8B9wE3JhiUiIqUiTqJIzR77BeBGd/8r0D65kEREpJTEacx+38xuAE4ARptZO5rJNOMiItJ4cX7hnw48A5zk7muBSuDKRKMSEZGSkbNE4e4bgT+bWU8z6xvtbtiycSIi0mzF6fV0kpm9BSwHZgLLgKeSDkxEREpDnKqnnwLjgDfdfQChrWJ6kkGJiEjpiJMoqtx9NVBmZubuTwCae0JEpJWI0+tpvZl1Ap4H/mBmq4CaZMMSEZFSEadE8SXC3E6XEKqcVhAG3omISCsQp9fThrTN2xKMRURESlC9icLM1gKZphM3wN29Z2JRiYhIychW9VRJmPyv7iu1PyczO8HM3jSzRdGEgpnOOd3MFprZAjPTqnkiIiUmW9XTSKDS3R9L32lmJwErgVezXdjMyoEbgOMJYzBeNrOp7r4w7ZwhwPeBce6+1sx2y+/bEBGRpGRLFL8EvpFh/9vAb4HxOa49Fljk7ksAzOxe4BRgYdo5/wrcEE0Ngruvihl387d1Q+5zpJbt27ezfPlytmzZUuxQmqWKigr69+9P27Ztix2KNDPZEkXv1C/5dO7+lpnFqXrqRxjFnbIcOLTOOfsCmNkLQDlhSvP/i3Ht5qtd57Bm9lP/CR/Oh+N+CL32LnZUzcLy5cvp0qULgwYNwgq1DnoL4e6sWbOG5cuXM3jw4GKHI81MtjaKDlmOdYxx7Uz/k+s2jrcBhgDHAGcCt5pZ910uZHaBmc02s9lV1VUxbl3C2rSDbz4HR/8/eOtxuGEsPPxd2PBBsSMreVu2bKFXr15KEnkwM3r16qXSmOQlW6J4ysyurrvTzK4i3hQey4EBadv9CW0bdc95yN23u/s7wJuExFGLu9/s7qPdfXSb8jhjBEtcRVc47gfwb3Nh1Lkw5w9w3cEwbQpsXlfs6EqakkT+9OwkX9kSxWXAMDN7y8zui15vAgcB341x7ZeBIWY2OFrDYiIwtc45fwOOBTCzSkJV1C7VXS1W593gpP+CyS/Dfp+H534F142EF66D7frLT0RKQ72Jwt03uPtphJXt7o1eX3T3U+sMwqvv81XAZOAx4A3gz+6+wMymmNnJ0WmPAWvMbCHwNHC5u69p3LfUDPXcC069LVRJ9RsFT/wQfnMIzLkLmntVWwvTuXPnYocgUnDmnmlMXenqNmCor1/2z2KHkax3noMnfwQrXoHK/WD8D2HoF6CVVx288cYb7L///kWNoXPnzmzcuLHWvurqasrLy4sUUcOUwjOU4jCzV9x9dD6f1ZKmpWjwUfCNaXD6XeA1cN9ZocFbSsb06dM59thj+epXv8qBBx5Y7HBEEtUCWoZbKDMYdnJou7jrS7BsZrEjKilX/30BC1d+0qTXHNa3Kz/64gGxz581axbz589Xd1Np8bKWKMys3My07GkxlbeBDrv0GJYSMHbsWCUJaRWylijcvTqah6mfu68oVFAiuTTkL/+kdOrUqdghiBREnKqnSuANM3sJ2JTa6e4TEotKRERKRpxEcU3iUYiISMmKs3DRtGgwXKpb1Wx3/yjZsERKU6pr7DHHHMMxxxxT3GBECiRn91gz+wowBzgbOAeYbWZfTjowEREpDXGqnq4Cxrj7hwBmtjvwOPBgkoGJiEhpiJMoylJJIrIaDdQrvI0fhrmgOlZCp97QqRI69grv23dp9aO2RSQ5cRLF42b2KJBapnQiYY4mKZR+o2HRU2F22UzK24WEkUocnSqh+0A4+nJo076wsYpIixMnUXwPOA04krDGxO+BvyQZlNRx5CXhtX0zbPoIPv0ofN3xfjVsWrPz/YcLYOMHMORzMGBMsaMXkWYuTq8nB/4cvaSY2naA7gPCK5tF0+CPE8I8USIijaS2BhERyUqJQqQBGrIexbp167jxxht3bE+fPp0vfOELTR7TnXfeyeTJk5v8uiIpShQijVRdXZ1xf91E0djriRRL1jYKMxsDnAUcBfQBNgPzgUeAu+OsdCeSiH9cCR/Ma9pr7nEgnBhvxprp06dz9dVX06dPH+bOncvChQt3OefKK69k8eLFjBw5kuOPP56TTjqJjRs3cuqppzJ//nxGjRrFH//4R8yMQYMGcd555/H4448zefJkxowZw7e//W1Wr15Nx44dueWWWxg6dCj3338/V199NeXl5XTr1o1nn30WgJUrV3LCCSewePFivvzlL3Pttdc26aOR1q3eRGFmDwNrgIeAXwGrgArCutbHAo+Y2bXu/nAhAhUpNbnWo7jmmmuYP38+c+fOBUJyefXVV1mwYAF9+/Zl3LhxvPDCCxx55JEAVFRU8PzzzwMwfvx4brrpJoYMGcLMmTO56KKLeOqpp5gyZQqPPfYY/fr1Y926dTvuNXfuXF599VXat2/Pfvvtx8UXX8yAATk6PYjElK1EcX6dgXYAW4BZ0esXZrZbYpGJZBPzL/8k5bMexdixY+nfvz8AI0eOZOnSpTsSxRlnnAGE+aRefPFFTjvttB2f27p1KwDjxo1j0qRJnH766UyYsHMC5/Hjx9OtWzcAhg0bxrvvvqtEIU2m3kSRNmXHhcA97r4+wzmrEoxNpKTlsx5F+/Y7B0CWl5dTVVW1y/Vqamro3r37jpJIuptuuomZM2fyyCOPMHLkyB3nZLuuSGPFacweBMwxs7vN7DMJxyPSYnTp0oUNGxrejNe1a1cGDx7M/fffD4C789prYaHJxYsXc+ihhzJlyhQqKytZtmxZk8YskknOROHuVwJDgD8BF5rZ22Y2xcwGJRybSLPWq1cvxo0bx/Dhw7n88ssb9Nk//elP3HbbbYwYMYIDDjiAhx56CIDLL7+cAw88kOHDh3P00UczYsSIJEIXqcXCwOsYJ5odAJwLfBF4FhgLPOru308uvF11GzDU1y/7ZyFv2fykRmaf9zgMPLTY0TSZN954g/3337/YYTRreoatl5m94u6jc5+5q5xTeJjZRcAk4BPgNuAH7r7VzMqARUBBE4WIiBRWnEkB+wMT3X1J+k53rzGzk5MJS6T5WLNmDePHj99l/7Rp0+jVq1cRIhJpWnESRd+6ScLM7nT3Se4+P6G4RDJyd6zE1t7o1atXxh5KpSZuNbNIXXF6PR2UvhFVOWnuaim4iooK1qxZo194eXB31qxZQ0VFRbFDkWYo28jsK4ArgS5m9nFqN+CEtgqRgurfvz/Lly9n9erVxQ6lWaqoqNgx2E+kIbJVPV1LmLrj54SEAYC7a8YyKYq2bds2eCS0iDRetkSxj7u/bWZ3AQekdqbqh9399YRjExGREpAtUVwJnA/ckOGYA0cnEpGIiJSUbHM9nR99Papw4YiISKnJ2evJzOaY2eVmtmchAhIRkdISp3vsaUBbYKqZvWRml5hZvzgXN7MTzOxNM1tkZldmOe9UM3Mzy2t4uYiIJCfOpICL3f1n7j4COA8YBbyb63NmVk5o3zgRGAacaWbDMpzXBfgOMLOBsYuISAHEWjPbzPqb2aXAnYRf+j+I8bGxwCJ3X+Lu24B7gVMynPefhK64W2JFLCIiBRWnjeIF4FGgE3C2u49y91/EuHY/IH2y/OXRvvRrHwwMyLWcqpldYGazzWx2VbUWZBERKaQ4cz19M885nTJNyLNj7oVoKpD/IcxMm5W73wzcDGGa8TxiERGRPGWbwuNMd78HOM7Mjqt73N2vy3Ht5UD6or39gZVp212A4cD0aBDfHoQG85PdfXbM+EVEJGHZShQ9oq+9MxyL81f9y8AQMxsMrAAmAl/dcYGwBndlatvMpgPfU5IQESkt2Qbc3Ri9fcTdZ6QfM7PDcl3Y3avMbDLwGFAO3O7uC8xsCjDb3ac2Im4RESmQOG0UNwKH1Nl3A6GbbFbu/iihITx931X1nHtMjFhERKTAsrVRjAUOB3qb2XfSDnUlDMATEZFWIFuJohOhDaENtdspNhBGa4uISCuQrY3iaeBpM7uj7lKoIiLSemSrevqVu18G/MrMdunl5O4TEo1MRERKQraqp/uir9cXIhARESlN2aqeZkVfp6X2mVk3oJ+7LyxAbCIiUgLizPU0zcy6mlkPYB5wt5n9MvnQRESkFMSZPbanu38CTAB+7+4jgc8lG5aIiJSKOImijZn1JnSJ/XvC8YiISImJkyh+CjwDvOfus8xsL+CdZMMSEZFSkXMKD3e/l7DoUGp7CZkXIBIRkRYoZ6Iws0rCEqiD0s939wuSC0tEREpFnEkBHwJmAM8D1cmGI01q1ULY/QBo37nYkYhIMxYnUXSKRmhLc9F5d7ByePgSeORS6D0U+h0C/UaF127DoFzzOopIPOaefQ0iM/s58LS7P16YkLLrNmCor1/2z2KHUfo2fQQr5sCKV3a+Nn8cjrWpgD4jdiaOfodAj8FgmVavFZGWwMxecffReX02RqJYC3QDPgW2EdbCdnfvmc8NG0uJIk/usHZplDSiBPL+XKjaEo536JGWOEZB30Ogc6bFDUWkOUo6UZRn2u/uRWmvUKJoQtXbYdUbaaWOObD6DfCacLz7wNrJo88IaNepuDGLSF4STRTRDSYCe7n7z8ysP7C7u7+Szw0bS4kiYVs3wvuv1U4e698Lx6wstG+kt3f03h/K4zR1iUgxJV2iuJ6wot3R7r6/mfUEHnP3MfncsLGUKIpg46pd2zu2rAvH2nSAviN3tnX0GwXd91R7h0iJaUyiiPOn4BHufoiZvQrg7h+bWbt8bibNVOfdYL8TwgtCe8fHS2onj1m3QPXWcLxjr13bOzr1Kl78ItIocRLFdjMrAxzAzHoBNYlGJaXNDHrtHV4HRaviVm8P4zbSq6zefoLon00oZezS3tGxaN+CiMQXp+rpHODLwGjgduB04Opoao+CU9VTM7J1Q4b2jmXhmJVnaO8YqvYOkYQUojH7AOAz0eY0d5+fz82aghJFM7fhQ1hZt71jfTjWtiP0GVk7eXQfqPYOkSaQSKIwswpge6obrJntA5wIvOvuU/MNtrGUKFqYHe0daYnj/dfT2jsqa1dZ9TsEOhZlCI9Is5ZUongG+Fd3f8vM9gZeJqyjvT/wgrv/IN+AG0OJohWo2garFtQeHLj6TXa0d/QYXKe94yBo26GoIYuUuqQSxTx3PzB6PwWodPeLzKw9MDt1rNCUKFqpLZ+EkeTp7R2frAjHrDxMfpiePHrvB2UZx4qKtEpJdY9NzyDHAb8CcPetZqZeT1JYFV1h8NHhlfLJ+7XbO+Y/AK/cEY617QR9D67d3tGtv9o7RPKQLVEsMLNrgBXAvsDjAGbWjTDfk0hxde0DXU+CoSeF7Zoa+Hhx7faOmTdB9bZwvNNutds6+h6s9g6RGLJVPXUCvgv0AW5z9znR/nHAEHe/s1BBplPVkzRI1Vb4sE57x0dv7jzec6/aVVZ7HKj2DmmREu8eW0qUKKTRtqyHlXXaOzasDMfK2uza3lG5r9o7pNlLqjH7b8DvgCfcvarOsT2BrwPL3f32fG6cLyUKScQnK2tPSbLyVdj6STjWrvOu7R1d+6m9Q5qVpBJFP+Ay4EvAh8BqoALYC3gPuMHd/5pXxI2gRCEFUVMDaxbVbu/4YB7UbA/HO+9eeyLEvgeHNT1ESlQhRmbvQ2ir2Ay86e4bYgZ2AvBroBy41d2vqXP8UuAbQBUhEZ3n7u9mu6YShRRN1VbvfrRmAAAVnklEQVT4YH7t5LHm7Z3He+1Tu8pq9+HQtqJ48YqkKck2imjBo7eA44HlhAF7Z7r7wrRzjgVmuvunZvYt4Bh3PyPbdZUopKRsXheqqXY0ls+GjR+GY2VtYY/hIWmMmhQaykWKJOlpxvM1Fljk7ksAzOxe4BRgR6Jw96fTzp8BnJVgPCJNr0N32PvY8IIwJcknK2uXOubeAy/fBoecDcf+B3TZvbgxizRQkomiH7AsbXs5cGiW888H/pFgPCLJM4Nu/cJr2Mlh3+a18MwvYdbvwqDAoy6Fw76tailpNsrinGRm7aJ2iobI1CUkYz2XmZ1FmMb8l/Ucv8DMZpvZ7KrqqkyniJSuDj3ghJ/BRTPDyPJpU+CGMbDgwVACESlxOROFmZ0EzAOeiLZHmtmDMa69HBiQtt0fWJnh+p8BfgCc7O5bM13I3W9299HuPrqN1iuQ5qpyHzjzHjjnIWjfFe6fBHecGNo2REpYnBLFFEKV0ToAd58LxCldvAwMMbPB0dKpE4Fa05Ob2cGEsRonu/uqhgQu0mztdQx881n44q9DF9xbjoUHLwxtGyIlKE6i2O7u6+rsy1lejgbpTQYeA94A/uzuC8xsiplFlbf8EugM3G9mc82saOtciBRUWXnoCXXxHBh3Ccz/K/xmFEz/BWz7tNjRidQSZynUOwiNzD8gDL77N6Cju1+QfHi7UvdYaZE+fgee/BEsfCiM+h5zflhnvMse0HmP8LV952JHKc1YouMooskBrwI+G+16jLBm9uZ8bthYShTSor37Ijz272FsRl3tuoSutV36RAkk7X2XPcL7zrsroUhGSSeKCe7+QK59haJEIS2ee5i4cOOHsOH9sM74hvdhwwew8YPwNbVdtWXXz7frkpY89qhdKklPLO06Ff57k6JJOlHMcfdDMtxwVD43bCwlCpFIKqHskkAyJJZMCWX8VXDUZYWPW4oikZHZZvY54ASgn5n9d9qhroBWuBMpNrMwMrxDd9htaP3nucOWdbUTyGP/DjN+G3paDToS9jwSOvcuXOzSrGQblLAKmA9sARak7d8AXJlkUCLShMzCoL8OPXYmlDbtYe7d8Nq98PKtYV/voSFpKHFIHXGqnircPUO5tThU9STShKq3w/uvwdLnYOnz8O5LsH1TONZ7/52JY9CR0KmyuLFKoyTdRrE38FNgGGE9CgDcfd98bthYShQiCUoljneeDYnjvRlKHC1E0oniOeAnwH8RxlGcC9S4+1X53LCxlChECqh6e1g2NlXiSE8cuw1Lq6oap8RR4pJOFK+4+ygzm+fuB0b7nnP3o/K5YWMpUYgU0Y7EkV7iiEaS10ocR0KnXsWNVWpJOlG8CIwDHgT+D1gB/Je775fPDRtLiUKkhFRvD4MDa5U4UonjgDolDiWOYko6URxKWGyoB6GtohvwC3d/IZ8bNpYShUgJq9oG78/d2caxbKYSR4ko+FKoZtbf3Zfnc8PGUqIQaUaqttUucaQnjt2H104cHXsWN9YWLrFEYWZjCCvVPe/uH5nZAcAVwHHu3j+vaBtJiUKkGduROFJtHDOhKpo2TokjUYkkCjP7OfAV4DVgMKGN4t+AXwC/dfeizIWsRCHSglRtg5Vz0to4UonD6iSOI5Q4GimpRLEQGOXum82sJ2F1uhHu/mb+oTaeEoVIC5ZKHO88F5LHsllKHE0kqURRazJAM5vr7iPzjLHJKFGItCJVW8NSsUufjxLHzGiCQ4M9hsOgo0LiGHi4EkcOSSWKdcBTqU3g2LRt3H1CPjdsLCUKkVZsR+JIL3FkSBx7HhHmtpIdkkoU47N90N2n5XPDxlKiEJEdqrbCilfSShzpiePAtMRxeKtPHAXvHltMShQiUq9U4khv46jeihKHEoWISGbbt+xa4kgljj4H1W7j6NC92NEmSolCRCSOHYkjNQCw9SSOgiQKM2vv7lvzuUlTUqIQkSazfQusmB2VONISh5XBHgdF3XGPClVVFd2KHW2jJD3X01jgNqCbuw80sxHAN9z94nxu2FhKFCKSmFTieCcqcSyfBdXbaieOwUfDwMOaXeJIOlHMAM4A/ubuB0f75rv78Hxu2FhKFCJSMNs3w/K0Ekd64ugzYmeJoxkkjsYkimxrZqeUufu7Zpa+rzqfm4mINCttO8Dgo8IL0hJHVOKY+Tt48Td1EkeqxNG1uLE3oTiJYllU/eRmVg5cDLyVbFgiIiUoY+J4eWeJo1biGFmnxNF8E0ecqqfdgOuAz0S7ngQmu/tHCceWkaqeRKRkpRLHjjaOl6Fme+3EMfhoGHBowRNH0m0UPd3947wiS4AShYg0G9s+rV3i2JE4yqFvWoljzyOgXadEQ0k6USwG3gTuAx5w9w353KipKFGISLO1I3GkShyzQ+IYfDR8/e+J3jrRxmx339vMjgAmAleb2VzgXne/N58bioi0Wu06wl7/El4QEsc9E2HT6uLGlUNZnJPc/UV3/w5wCPAJ8KdEoxIRaQ3adWwWjdw5E4WZdTazr5nZ34FZwGrgiMQjExGRkhCnRDEfOAy41t33cffL3H1mnIub2Qlm9qaZLTKzKzMcb29m90XHZ5rZoAZFLyLSEtRUhdX9SlSccRR7uXtNQy8cjbm4ATgeWA68bGZT3X1h2mnnA2vdfR8zm0hYj/uMht5LRKTZKm8PH70FP+kdpj7vtBt0Tr12h069w9fUvk67hX3lcX59N41672Rmv3L3y4C/mtkuXaNirHA3Fljk7kui690LnAKkJ4pTgB9H7/8CXG9m5s1tSlsRkXwdf3XoJrtpNWxcBRs/DO9XvgobV8O2TB1NLSz9Wl8iSU80HXtBWXmjQsyWku6Lvl6f57X7AcvStpcDh9Z3jrtXmdl6oBdQlMF8IiIF160/jD63/uPbNoUEsml1SCIbV0Xbq3a+XzYzfK3avOvnrQw6VjYqxHoThbvPit7u7+61koWZTQZyLYVqGfbVLSnEOQczuwC4INrcambzc9y7tahESTVFz2InPYud9CwAWAewX76fjlPJdR67lirOz7CvruXAgLTt/sDKes5ZbmZtgG7ALqPA3f1m4GYAM5ud76CRlkbPYic9i530LHbSs9jJzGbn+9lsbRRnEAbZDTazB9IOdSFKTzm8DAwxs8HAiuhaX61zzlTg68BLwKnAU2qfEBEpLdlKFLOANYSSwA1p+zcAr+a6cNTmMBl4DCgHbnf3BWY2BZjt7lMJCyLdZWaLCCWJifl9GyIikpRsbRTvAO8QZovNi7s/CjxaZ99Vae+3AKc18LI35xtPC6RnsZOexU56FjvpWeyU97Ood1JAM3vG3f/FzNZSu4HZAHf3nvneVEREmo9siaLM3WuigXO7cHetcici0grUO4VH2mjsAUB5lBgOB74JJDtxOpr+I12MZ3GpmS00s9fNbJqZ7VmMOAsh17NIO+9UM3Mza7E9XuI8CzM7Pfq3scDM7i50jIUS4//IQDN72sxejf6ffL4YcSbNzG43s1X1DSGw4LroOb1uZofEurC7Z30Bc4G2wN7AEuA3wMO5PteYF6HxezGwF9AOeA0YVueci4CbovcTgfuSjKlYr5jP4ligY/T+W635WUTndQGeBWYAo4sddxH/XQwhdDzpEW3vVuy4i/gsbga+Fb0fBiwtdtwJPYujCbN8z6/n+OeBfxCaEA4DZsa5bpxJAWvcfTswAfhfd7+YMKI6STum/3D3bUBq+o90pwC/j97/BRhvZpkG8DV3OZ+Fuz/t7p9GmzMIPdVaojj/LgD+E7gW2FLI4AoszrP4V+AGd18L4O6rChxjocR5Fg6k5vPuxq5juloEd3+WDGPR0pwC/MGDGUB3M+uT67pxEkWVmZ0GnA08HO1rG+NzjZFp+o+6yanW9B9AavqPlibOs0h3PuEvhpYo57Mws4OBAe7+MC1bnH8X+wL7mtkLZjbDzE4oWHSFFedZ/Bg4y8yWE3piXlyY0EpOQ3+fAPESxXmEqo1r3X1JNIDunrxCjK/Jpv9oAWJ/n2Z2FjAa+GWiERVP1mdhZmXA/wCXFSyi4onz76INofrpGOBM4FYz655wXMUQ51mcCdzp7v0J1S93Rf9eWpu8fm/mfFDuPh/4DjDbzIYCy9z9pw2Pr0EaMv0H2ab/aAHiPAvM7DPAD4CT3X1rgWIrtFzPogswHJhuZksJdbBTW2iDdtz/Iw+5+3YP46LeJCSOlibOszgf+DOAu78EVBDmgWptYv0+qSvOCndHAYsIo6hvB94ys3F5BhnXjuk/zKwdobF6ap1zUtN/QMue/iPns4iqW35HSBIttR4acjwLd1/v7pXuPsjdBxHaa05297znuClhcf6P/I1QG4CZVRKqopYUNMrCiPMs3gPGA5jZ/oREUdoLVSdjKnBO1PvpMGC9u7+f81MxWtFnk9aDANifMAVH0q33nwfeIvRm+EG0bwrhPz6EH/T9hCQ2i7DAUtF7HRTpWTwJfEjooTYXmFrsmIv1LOqcO50W2usp5r8LA/6bsAbMPGBisWMu4rMYBrxA6BE1F/hssWNO6DncA7wPbCeUHs4HLgQuTPs3cUP0nObF/f9R74C7FDN73d0PyrVPRERapjjTjM8xs98Bd0XbXyPGpIAiItIyxClRVBAas48kFFueBX7jYUI/ERFp4bImCjM7kDAie4G7v12wqEREpGTU2+vJzP6d0Gvia8ATZnZewaISEZGSkW322AXAWHffZGa9gUfdfUxBoxMRkaLLNo5iq7tvAnD31TnOlWbGzKrNbG7aa1CWcwfVNxtlA+85PZrh87VoWokGL/ZuZhea2TnR+0lm1jft2K1mNqyJ43zZzEbG+MwlZtYxj3v9r5kdHb2fHM3q6dG4h4Zea78o9rlm9oaZNemiPWZ2cmpmVjPrbWHW5lfN7CgzezTbqO9sP7csn3nSzHo03Xcg+cpWolgHPJXaJAzcSW3j7hMSj04SY2Yb3b1zzHMHEWYMHt7Ie04Hvufus83sAuAL7n5yU1yvMXFlu66ZnQt81d2Pz/GZpYQ+6R814D49CSX1w6Ltg4G17Bz/Efta0ecfA25094ei7QPdfV5DrtGAe00ETnT3r+c8edfPTifGz83Mvg709+RngpAcspUSvkIYmHEDcH2d7RuyfE6aqajk8JyZzYleR2Q45wAzmxX91fq6mQ2J9p+Vtv93Vs+CV2meBfaJPjs++st0noX59NtH+6+xnets/Fe078dm9j0zO5Uwr9Wfont2iP6aHm1m3zKza9NinmRmv8kzzpdImzTNzH5rZrMtrO9wdbTvO0Bf4Gkzezra91kzeyl6jvebWaakfCrwf6kNd3/V3ZfmiCebPoRBVqnrzYtimWRmD5nZ/0UlpR+lfT8Zn4eF9R3mRKWqaWnXuT4qYV0LfD7t2S9NlYLM7JzoZ/aamd0V7avv53aSmT2YFs/xZvZAtDmVMEeTFFuxRxLqVZwXUM3OkdwPRvs6AhXR+yFEI/CBQUTz2xPWI/la9L4d0IEwWv/vQNto/43AORnuOZ1oJChwOXAfYYT9MmDfaP8fgEuAnoS5iVKl3u7R1x8T/hqtdb30baA3Ydrp1P5/ELp35xPnJcDP0o71jL6WR+cdFG0vBSqj95WERNgp2r4CuCrDfX4PfDHD/h3XauDP9FzCLMr/AL6b9swmEUbr9op+XvOj55TxeUTPbxkwuM73PAm4vu779JiBA6KfW2Wdz2b8uRFqK/4J9I62705/JsDbQK9i/39p7a84A+6kZdrs7nXr3tsCqb8YqwlzA9X1EvADM+sPPODub5vZeGAU8LKFJUE6APXNOfUnM9tM+MVyMbAf8I67vxUd/z3wbUIpdgthxtNH2DnFfU7uvtrMlliYy+bt6B4vRNdtSJydCAkhfRWw06NqszaEv+CHAa/X+exh0f4Xovu0Izy3uvrQhPMNufsdUfXTCYR1B75pZiOiw0+4+xqA6C/2I4EqMj+Pw4BnPUwkiLs3ZLLN44C/eFRtluuz7u5RqeMsM7uDsIrmOWmnrCKU1tY0IAZpYkoUku67hDmjRhCqJXcZVOnud5vZTOAk4DEz+wbhr8Lfu/v3Y9zja55WN21mGdcQcfcqMxtLmMhtIjCZ8EsorvuA0wl/rT4Y/UJqUJyEeYGuIVS1TrAwxf73gDHuvtbM7iSUiOoywi/mXNUmm+v5fL2iX6YHAyvdfZflPN19JWHyztstdEBItSvVbYx06vm5mdnJGc6PHWIen72DULLZAtzvYX2ZlArCc5Iiit2TKVVvLC1aN+B9D+uln034a7oWM9sLWOLu1xHqkA8CpgGnmtlu0Tk9Lf663f8EBpnZPtH22cAzUZ1+N3d/lFD9k6nn0QbC1OKZPAB8iVDHfV+0r0FxeljZ8T+AwyzMONoV2ASsN7PdgRPriWUGMC71PZlZRzPLVDp7g6idJi53P9fdR2ZKElG7Qtvo/R6EqqYV0eHjo++3A+G5vED9z+Ml4F+ixJhqdI9rGqHU1SvLZ2v93KLktpLwrO9M+34M2INQ+pQiijPN+Fgzm0cowmNmI1INg9Li3Ah83cxmEKqdNmU45wxgvpnNBYYSllVcSPhP/riZvQ48QahWycnDVDDnAvdH/85qgJsIv0gejq73DKG0U9edwE2pBtU6111LmDV1T3efFe1rcJzuvhn4FaF+/TXCPGcLCH+1v5B26s3AP8zsaQ/dyScB90T3mUF4VnU9QlhUCAiN4hZWYOsPvG5mt2aLLYPPEn42rwGPAZe7+wfRsecJ87XNBf7q7rPrex5R/BcAD0TXuq/ujerj7guAnxKS/WuE2WvrupNdf25/Iqx1szDtvFHAjDolDCmCOHM9zSD8cvibux8c7ZvvjewqKSJgZs8TugmvS/AekwiNx5OTukdjmdn1wKvuflvavl8TpsyfVrzIBOJVPZW5+7t19lUnEYxIK3QZMLDYQRSTmb1CqML8Y51D85UkSkOcEsVfgV8QqgPGEHqqjHP305IPT0REii1OotgNuA74TLTrSWCyN3DUqIiINE85E4WIiLRuOcdRmNktZOgX7e4XJBKRiIiUlDgD7p5Me18BfJkwvF9ERFqBBlc9mVkZYdTp+GRCEhGRUpLPGhODgbijbkVEpJmL00axlp1tFGXAx8CVSQYlIiKlI2vVUzTXygB2zhdT4+omJSLSqmSteoqSwoPuXh29lCRERFqZOG0Us8zskNyniYhIS5Rtzew20ZoA8wgrYS0mzCZqhMKGkoeISCuQrTF7FmFlry8VKBYRESlB2RKFAbj74gLFIiIiJShbouhtZpfWd9DdMy1IIiIiLUy2RFEOdCYqWYiISOuUrTF7jhqsRUQkW/dYlSRERCRriaKnu39c4HhERKTEaOEiERHJKp/ZY0VEpBVRohARkayUKEREJCslChERyUqJQkREsvr/tBD1XEF4+l8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a19477b70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot LR and RF model ROC curves\n",
    "sns.mpl.pyplot.plot(fpr_lr, tpr_lr,label=\"lr\")\n",
    "sns.mpl.pyplot.plot(fpr_lr,thresholds_lr, label=\"lr_thresh\")\n",
    "sns.mpl.pyplot.xlim([0, 1])\n",
    "sns.mpl.pyplot.ylim([0, 1.05])\n",
    "sns.mpl.pyplot.legend(loc=\"center\")\n",
    "sns.mpl.pyplot.xlabel('False Positive Rate (1 - Specificity)')\n",
    "sns.mpl.pyplot.ylabel('True Positive Rate (Sensitivity) or Class 1 Threshold Probability')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the threshold that satisfies our criteria is ~0.7. Let's use this new threshold to generate our predictions and see what our new confusion matrix looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   class_0  class_1  predicted  actual  predicted_07\n",
      "0    0.018    0.982        1.0     1.0           1.0\n",
      "1    0.111    0.889        1.0     0.0           1.0\n",
      "2    0.011    0.989        1.0     1.0           1.0\n",
      "3    0.094    0.906        1.0     1.0           1.0\n",
      "4    0.010    0.990        1.0     1.0           1.0\n",
      "Confusion matrix at original 0.5 threshold:\n",
      " [[21  6]\n",
      " [ 6 60]] \n",
      "\n",
      "Classification Report at original 0.5 threshold:\n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.78      0.78      0.78        27\n",
      "         1.0       0.91      0.91      0.91        66\n",
      "\n",
      "   micro avg       0.87      0.87      0.87        93\n",
      "   macro avg       0.84      0.84      0.84        93\n",
      "weighted avg       0.87      0.87      0.87        93\n",
      " \n",
      "\n",
      "Confusion matrix at 0.7 threshold:\n",
      " [[24  3]\n",
      " [11 55]] \n",
      "\n",
      "Classification Report at 0.7 threshold:\n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.69      0.89      0.77        27\n",
      "         1.0       0.95      0.83      0.89        66\n",
      "\n",
      "   micro avg       0.85      0.85      0.85        93\n",
      "   macro avg       0.82      0.86      0.83        93\n",
      "weighted avg       0.87      0.85      0.85        93\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_test_lr_df[\"predicted_07\"] = (y_test_lr_df.class_1 > 0.7).astype(float)\n",
    "print(y_test_lr_df.head())\n",
    "print(\"Confusion matrix at original 0.5 threshold:\\n\",metrics.confusion_matrix(y_test_lr_df.actual,\n",
    "                                                                      y_test_lr_df.predicted),\"\\n\")\n",
    "print(\"Classification Report at original 0.5 threshold:\\n\", metrics.classification_report(y_test_lr_df.actual,\n",
    "                                                                                          y_test_lr_df.predicted),\"\\n\")\n",
    "print(\"Confusion matrix at 0.7 threshold:\\n\",metrics.confusion_matrix(y_test_lr_df.actual,\n",
    "                                                                      y_test_lr_df.predicted_07),\"\\n\")\n",
    "print(\"Classification Report at 0.7 threshold:\\n\", metrics.classification_report(y_test_lr_df.actual,\n",
    "                                                                                 y_test_lr_df.predicted_07))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, we've now increased our precision for class 1 (we are more certain someone has a back problem, when our classifier says you have a back problem), but have also lowered our recall for class 1 (now, more people with back problems will go un-identified by our classifier). The opposite occurs for precision and recall for class 0 as a result.\n",
    "\n",
    "**So, depending on what you want to maximize or minimize for your machine learning application, your decision threshold for your (completely trained) classifier will differ.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**One final important point about computing the AUC, you have to use the predicted probabilities, not the class labels themselves when computing an ROC curve or AUC.**\n",
    "\n",
    "If you use y_pred_class, it will not give you an error, rather it will interpret the ones and zeros as predicted probabilities of 100% and 0%, and thus will give you incorrect results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrong way to calculate LR model AUC:  0.8434343434343434\n",
      "Wrong way to calculate RF model AUC:  0.8282828282828283\n"
     ]
    }
   ],
   "source": [
    "# calculate AUC using y_pred_class (producing incorrect results)\n",
    "print(\"Wrong way to calculate LR model AUC: \",metrics.roc_auc_score(y_test, predictions_lr))\n",
    "print(\"Wrong way to calculate RF model AUC: \",metrics.roc_auc_score(y_test, predictions_rf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a final way to compare the two models, let's see if we can look at each of their predicted probabilites on the test set as a function of the actual category (the target):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x1a1b85ca90>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x1a1b80c198>],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAETCAYAAADecgZGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEOxJREFUeJzt3X2MZXV9x/H3x10QC1UeRQSnS9vVik3RPiiNadJKrVis8IemPpU1JcW0tfEp6mpJq9a2QJNqm1grEcpqVTRWBcVqLYKtUXnGh3UrqxRWRAUVUEzVIt/+cc92h90ZZnbmnnPP7O/9SiZ777nnzv3MufP77JnzdFNVSJLa8oBZB5AkDc/yl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/GckyaFJ3p/k+0luTvKcReZLkrOTfLv7OidJhs4r9SXJC5NcneSHSS5YYt6XJPlGkruSnJ/kgQPF3OdY/rPzJuBHwJHAc4E3J3nMAvOdAZwKHA/8AvA04AVDhZQGcCvweuD8+5spyVOAzcCJwAbgp4HX9h1uXxXP8B1ekgOBO4Cfr6obumlvB75WVZt3m/dTwAVVdW53/3TgD6rqhIFjS71K8nrgmKp6/iKPvxO4qape3d0/EXhHVT1suJT7Dtf8Z+ORwI93Fn/ns8BCa/6P6R5baj5pX7fQWDgyyWEzyrOmWf6zcRBw127T7gJ+chnz3gUc5HZ/NWihsQALjxstwfKfjbuBB+827cHA95Yx74OBu8vtdWrPQmMBFh43WoLlPxs3AOuTbJw37Xhg6wLzbu0eW2o+aV+30Fj4ZlV9e0Z51jTLfwaq6vvA+4DXJTkwyROBU4C3LzD724CXJjk6ycOBlwEXDBZW6lmS9UkOANYB65IckGT9ArO+DTg9yXFJDgHOxLGwYpb/7PwR8CDgNuBdwB9W1dYkv5bk7nnzvQX4IPB54AvAJd00aV9xJvA/TA7jfF53+8wkc0nuTjIHUFUfAc4BLgNu7r7+fDaR1z4P9ZSkBrnmL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUoIWOpe3N4YcfXhs2bBjyJdWAa6655ltVdcSsc+wNx4L6sDdjYdDy37BhA1dfffWQL6kGJLl51hn2lmNBfdibseBmH0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDBj3JS5K0y4bNl6zoeTeddfKqX9s1f0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDVp2+SdZl+S6JB/q7h+b5Iok25O8O8n+/cWUJE3T3qz5vwjYNu/+2cAbqmojcAdw+jSDSZL6s6zyT3IMcDLw1u5+gCcB7+1m2QKc2kdASdL0LXfN/43AK4B7u/uHAXdW1T3d/VuAo6ecTZLUkyXLP8nTgNuq6pr5kxeYtRZ5/hndfoHbd+zYscKY0trnWNCYLGfN/4nA05PcBFzIZHPPG4GDk6zv5jkGuHWhJ1fVuVW1saqOmJubm0JkaW1yLGhMliz/qnpVVR1TVRuAZwEfr6rnApcBz+hm2wRc1FtKSdJUreY4/1cCL03yZSb7AM6bTiRJUt/WLz3LLlV1OXB5d/tG4PHTjyRJ6ptn+EpSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDVoyfJPckCSK5N8NsnWJK/tph+b5Iok25O8O8n+/ceVJE3Dctb8fwg8qaqOBx4LnJTkBOBs4A1VtRG4Azi9v5iSpGlasvxr4u7u7n7dVwFPAt7bTd8CnNpLQknS1C1rm3+SdUmuB24DPgZ8Bbizqu7pZrkFOHqR557RbRq6fceOHdPILK1JjgWNybLKv6p+XFWPBY4BHg88eqHZFnnuuVW1saqOmJubW3lSaY1zLGhM9upon6q6E7gcOAE4OMn67qFjgFunG02S1JflHO1zRJKDu9sPAn4T2AZcBjyjm20TcFFfISVJ07V+6Vk4CtiSZB2T/yzeU1UfSvJF4MIkrweuA87rMackaYqWLP+q+hzwuAWm38hk+78kaY3xDF9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGLVn+SR6R5LIk25JsTfKibvqhST6WZHv37yH9x5UkTcNy1vzvAV5WVY8GTgD+OMlxwGbg0qraCFza3ZckrQFLln9Vfb2qru1ufw/YBhwNnAJs6WbbApzaV0hJ0nTt1Tb/JBuAxwFXAEdW1ddh8h8E8NBFnnNGt2no9h07dqwurbSGORY0Jssu/yQHAf8CvLiqvrvc51XVuVW1saqOmJubW0lGaZ/gWNCYLKv8k+zHpPjfUVXv6yZ/M8lR3eNHAbf1E1GSNG3LOdonwHnAtqr623kPXQxs6m5vAi6afjxJUh/WL2OeJwK/B3w+yfXdtFcDZwHvSXI6sAN4Zj8RJUnTtmT5V9UngSzy8InTjSNJGoJn+EpSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQUuWf5Lzk9yW5Avzph2a5GNJtnf/HtJvTEnSNC1nzf8C4KTdpm0GLq2qjcCl3X1J0hqxZPlX1X8A39lt8inAlu72FuDUKeeSJPVopdv8j6yqrwN0/z50epEkSX3rfYdvkjO6fQO379ixo++Xk0bLsaAxWWn5fzPJUQDdv7ctNmNVnVtVG6vqiLm5uRW+nLT2ORY0Jist/4uBTd3tTcBF04kjSRrCcg71fBfwaeBRSW5JcjpwFvDkJNuBJ3f3JUlrxPqlZqiqZy/y0IlTziJJGohn+EpSgyx/SWqQ5S9JDbL8JalBS+7wHcqGzZfs1fw3nXVyT0kkad/nmr8kNcjyl6QGWf6S1CDLX5IaNJodvn1zh7Ik7eKavyQ1yPKXpAZZ/pLUoGa2+UtSn/Z2v+KsueYvSQ2y/CWpQZa/JDXI8pekBq3ZHb5rbeeKJI2Ja/6S1CDLX5IaZPlLUoPW7DZ/Se1Y6T6+lVygsZX9ia75S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhrkSV5asb09GWYlJ9xo7wx5MtRKtXIS1di55i9JDbL8JalBlr8kNcjyl6QGucN3EX3vzHSnl+7P0L8fK3m9tbAD33G2uFWt+Sc5KcmXknw5yeZphZIk9WvF5Z9kHfAm4KnAccCzkxw3rWCSpP6sZs3/8cCXq+rGqvoRcCFwynRiSZL6tJryPxr46rz7t3TTJEkjt5odvllgWu0xU3IG8HLgYOAHSbYu8v0OB761ijzTsqIcOXscOXowtRyrXEb3l+OnVvWdB7KvjYUefudXlGNAY8lBzl40y7LHQqr26OvlPTH5VeA1VfWU7v6rAKrqr1f4/a6uql9eUZgpMoc5Zm0sP6s5xpkDppNlNZt9rgI2Jjk2yf7As4CLVxNGkjSMFW/2qap7krwQ+CiwDji/qhb7M1aSNCKrOsmrqj4MfHhKWc6d0vdZLXPclzmGN5af1Rz3NZYcMIUsK97mL0lau7y2jyQ1yPKXpAZZ/pLUoJlc1TNJmFwe4mgmJ4bdClxZM9wBkeRY4HHAF6vqv2bw+g8BTuK+y+SjVXXn0FnmZZrpMmnBGMdCl2tm771jYRiDr/kn+S1gO/Aa4LeBk4HXAtu7x4bK8YF5t08BPg78DnBRkucPlaN7/dOAa4FfB34COBD4DeCa7rGhcoxpmTwkye8meWmSl3S3Dx4yQ9/GMha6LKN47x0Li+aZ/nioqkG/gG3AhgWmHwtsGzDHdfNufwo4trt9OPDZgZfJl4CDF5h+CHBDa8sEOA34CvBm4Mzu6x+7aacN+d70/HOOYiyM7L13LOyZpZfxMIvNPuuZXARud18D9hswx/w/q9dX1X8DVNW3ktw7YA6YXCdpoT/z72Xhayj1ZSzL5E+BX6rd/sxPcghwBfC2AbP0aSxjAcbz3jsW9tTLeJhF+Z8PXJXkQnZdFfQRTC4Pcd6AOY5P8l0mv1APTPKwqvpGd6mKdQPmAPhL4Nok/8auZTIHPBn4iwFzjGWZjKUA+jaWsQDjee8dC3vqZTzM5CSv7kNfns5kh06YrP1cXFVfHDzMbrrtaI+uqk8P/LqHAE/hvsvko1V1x5A5FjL0MkmyCfgzYMECqKoLhsgxhDGPBZjNeHAs7PGavYyHmZ7hm+RQoMbwpmpcxlwAfXAs6P70MR5mcbTPXJILk9zGZHvVlUlu66ZtGDDHI7rX/M8kr06y37zHPnB/z+0hy+/Pu310kkuT3JHkU0keOWCO7yR5a5ITu0MQZ6b7pb6s+7oUuGxfK8axjIUuyyjGg2NhYX2Mh1mc5PVu4P3AUVW1sao2AkcBH2DyUZBDOR+4HPiT7vU/keSw7rGhPxzkhfNuvwF4D3AY8DdM9vAP5XbgeuB1wC1J/i7JCQO+PgBJHpvkM0zen7OZLIdPJPlMkl8cOk+PxjIWYDzjwbGwm97Gw5CHLHWbmLav5LEecly/2/3nAVuBnwGuHXiZXHs/ua6bUY454BVMjrm+EfirId8b4AkLTD+BgQ+z6/nnHMVY2LnMd7s/k/HgWFj4veljPMziaJ9rkvwDsIX7HuGwCbhuwBz7JTmgqn4AUFX/nOQbTD6f4MABcwAck+TvmWzLOyLJflX1vztzDpjj//+8raodwDnAOUkexeQIlKEcWFVX7D6xqj6TZOj3pk9jGQswnvHgWNhTL+NhFuV/GnA6kzMZd+68+CrwQYY9vO2twBOAT+ycUFX/nuSZTN7oIb183u2rgYOAO5I8jGE/He2yhSZW1ZeYvF9D+dcklzA5fnl+KZ4GfGTAHH0by1iA8YwHx8KeehkPXs9fo5TkqcAp7HkI5LQ+PEhaM/oYD6Mq/yRPq6oPmWOXsWQZS45WjGl5jyWLOaZrbJd0/pVZB+iMJQeMJ8sociQ5Y9YZBjKK5d0ZSxZz7GY142FWZ/j+HLv+hNl5ydaLq2pbiznGlGUsORaT5AVV9ZZZ55iWMS3vsWQxx/KtZjzM4iSvVzI5hjnAlcBV3e13JdncWo4xZRlLjiX8aNYBpmVMy3ssWcyx11Y8HgZf809yA/CYeYdv7Zy+P7C1Jie6NJNjTFnGkuP+JNlRVXOzzjENY1reY8lijr2zmvEwi0M97wUeDty82/SjusdayzGmLKPIkeRziz0EHDlUjgGMYnmPLIs5dtPXeJhF+b8YuDTJdu57hbqf5b6ndreSY0xZxpLjSCYXsdr92iVh8sEa+4qxLO8xZTHHnnoZD7Pa4fsAdn1u6c5jVq+qqh+3mGNMWcaQI8l5wD9V1ScXeOydVfWcobL0bQzLe2xZzLFHjl7Gw6iO85ckDWNsx/lLkgZg+UtSgyx/SWqQ5S9JDbL8JalB/wcgh+I5Z/z0fgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a1b6bf390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAETCAYAAADecgZGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEipJREFUeJzt3XuQZGV9xvHvI4uXgArKiis6WWJWo6YCGoOkrKQ0RMVLglbFSjQKVqislUjKW5lsCBXRmARMlEqqjHEVAhqvFW8oxksIaiwVBURl3eiqwRFEFhUvWFED/PJHn5Vh6WFmp7tPn9n3+6mamu7TZ+Y8c3rfZ8+ceft0qgpJUlvuMO8AkqT+Wf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZb/nCS5R5J3Jvlhkq8lefoy6yXJmUm+3X28PEn6zivNSpJTklyS5MdJzl1h3ecn+WaS7yU5J8mdeoq537H85+dVwE+Aw4HfB16d5CFj1tsKPBk4Cvgl4EnAs/sKKfXgG8DLgHNub6UkjwO2AccBm4GfA14y63D7q/gK3/4lOQi4HvjFqvpSt+wNwNVVtW2vdT8OnFtV27v7JwN/WFXH9hxbmqkkLwPuW1XPWubxNwFXVtWp3f3jgDdW1b37S7n/8Mh/Ph4A3LSn+DufBcYd+T+ke2yl9aT93bixcHiSe84pz7pm+c/HwcD39lr2PeCuq1j3e8DBnvdXg8aNBRg/brQCy38+bgDutteyuwE/WMW6dwNuKM/XqT3jxgKMHzdageU/H18CNiTZsmTZUcCOMevu6B5baT1pfzduLFxbVd+eU551zfKfg6r6IfAO4KVJDkrySOAE4A1jVn898IIkRyS5D/BC4NzewkozlmRDkjsDBwAHJLlzkg1jVn09cHKSByc5FDgNx8KaWf7z88fAXYDdwJuBP6qqHUl+LckNS9Z7DfAe4PPAFcAF3TJpf3Ea8L+MpnE+o7t9WpKFJDckWQCoqvcDLwcuAr7Wfbx4PpHXP6d6SlKDPPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWrQuLm0M3PYYYfV5s2b+9ykGnDppZd+q6o2zjvHvnAsaFZWOx56Lf/NmzdzySWX9LlJNSDJ1+adYV85FjQrqx0PnvaRpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNajXF3lJkpa3edsFq1rvyjOeOPG2PPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNWjF8k9y5ySfSvLZJDuSvKRbfmSSi5PsSvLWJHecfVxJ0jSs5sj/x8BvVNVRwNHA8UmOBc4EzqqqLcD1wMmziylJmqYVy79GbujuHth9FPAbwL91y88DnjyThJKkqVvVOf8kByS5HNgNfAj4CvDdqrqxW+Uq4IjZRJQkTduqyr+qbqqqo4H7AscADxq32rivTbK1+7vAdYuLi2tPKq1zjgUNyT7N9qmq7wIfBo4FDkmy56qg9wW+sczXbK+qLVW1cWFhYZKs0rrmWNCQrGa2z8Ykh3S37wL8JrATuAj4nW61k4B3zyqkJGm6VnM9/03AeUkOYPSfxduq6r1JvgC8JcnLgM8AZ88wpyRpilYs/6r6HPDQMcu/yuj8vyRpnfEVvpLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1aMXyT3K/JBcl2ZlkR5LndstPT3J1ksu7jyfMPq4kaRo2rGKdG4EXVtVlSe4KXJrkQ91jZ1XV388uniRpFlYs/6q6Brimu/2DJDuBI2YdTJI0O/t0zj/JZuChwMXdolOSfC7JOUkOnXI2SdKMrLr8kxwMvB14XlV9H3g1cH/gaEa/Gbxima/bmmRXkusWFxenEFlanxwLGpJVlX+SAxkV/xur6h0AVXVtVd1UVTcDrwWOGfe1VbW9qrZU1caFhYVp5ZbWHceChmQ1s30CnA3srKpXLlm+aclqTwGumH48SdIsrGa2zyOBZwKfT3J5t+xU4GlJjgYKuBJ49kwSSpKmbjWzfT4GZMxD75t+HElSH3yFryQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUErln+S+yW5KMnOJDuSPLdbfo8kH0qyq/t86OzjSpKmYTVH/jcCL6yqBwHHAs9J8mBgG3BhVW0BLuzuS5LWgRXLv6quqarLuts/AHYCRwAnAOd1q50HPHlWISVJ07VP5/yTbAYeClwMHF5V18DoPwjgXst8zdbu1NB1i4uLk6WV1jHHgoZk1eWf5GDg7cDzqur7q/26qtpeVVuqauPCwsJaMkr7BceChmRV5Z/kQEbF/8aqeke3+Nokm7rHNwG7ZxNRkjRtq5ntE+BsYGdVvXLJQ+cDJ3W3TwLePf14kqRZ2LCKdR4JPBP4fJLLu2WnAmcAb0tyMrAIPHU2ESVJ07Zi+VfVx4As8/Bx040jSeqDr/CVpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUErln+Sc5LsTnLFkmWnJ7k6yeXdxxNmG1OSNE2rOfI/Fzh+zPKzquro7uN9040lSZqlFcu/qj4KfKeHLJKknkxyzv+UJJ/rTgsdutxKSbYm2ZXkusXFxQk2J61vjgUNyVrL/9XA/YGjgWuAVyy3YlVtr6otVbVxYWFhjZuT1j/HgoZkTeVfVddW1U1VdTPwWuCY6caSJM3Smso/yaYld58CXLHcupKk4dmw0gpJ3gw8CjgsyVXAi4FHJTkaKOBK4NkzzChJmrIVy7+qnjZm8dkzyCJJ6omv8JWkBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQSuWf5JzkuxOcsWSZfdI8qEku7rPh842piRpmlZz5H8ucPxey7YBF1bVFuDC7r4kaZ1Ysfyr6qPAd/ZafAJwXnf7PODJU84lSZqhtZ7zP7yqrgHoPt9ruRWTbO1OD123uLi4xs1J659jQUMy8z/4VtX2qtpSVRsXFhZmvTlpsBwLGpK1lv+1STYBdJ93Ty+SJGnW1lr+5wMndbdPAt49nTiSpD6sZqrnm4FPAA9MclWSk4EzgMck2QU8prsvSVonNqy0QlU9bZmHjptyFklST3yFryQ1aMUj/75s3nbBPq1/5RlPnFESSZqufe23PnjkL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktSgDZN8cZIrgR8ANwE3VtXDpxFKkjRbE5V/59FV9a0pfB9JUk887SNJDZr0yL+ADyYp4DVVtX3vFZJsBV4EHLJx48YJN7d2m7ddsE/rX3nGE2eURK0ayljQylbbF+u5JyY98n9kVT0MeDzwnCS/vvcKVbW9qrZU1caFhYUJNyetX44FDclE5V9V3+g+7wbeCRwzjVCSpNlac/knOSjJXffcBh4LXDGtYJKk2ZnknP/hwDuT7Pk+b6qq908llSRpptZc/lX1VeCoKWaRJPXEqZ6S1KBpvMhrLvZ16qakYZnmGF7tlEt74xYe+UtSgyx/SWqQ5S9JDbL8JalBlr8kNWjdzvaR1K8hX+zMWTz7ziN/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CCnei5jvb/n76ynvg3t55W0bzzyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkLN95mS9X4hqvc+G0uys93/b+2I9/6we+UtSgyx/SWrQROWf5PgkX0zy5STbphVKkjRbay7/JAcArwIeDzwYeFqSB08rmCRpdiY58j8G+HJVfbWqfgK8BThhOrEkSbM0SfkfAXx9yf2rumWSpIGbZKpnxiyr26yUbAVeBBwC/CjJjiUPHwZ8a4IM0zRRlpw5xST70X7ZYwr75/Zy/OzE370HrYyFKTPLGDlz8vGQqtv09eo2nvwqcHpVPa67/+cAVfW3+/A9Lqmqh68pwJSZZbyhZBlKjlkZ0s9nlvH2tyyTnPb5NLAlyZFJ7gj8HnD+JGEkSf1Y82mfqroxySnAB4ADgHOqascKXyZJGoCJLu9QVe8D3jfBt9g+yfanzCzjDSXLUHLMypB+PrOMt19lWfM5f0nS+uXlHSSpQZa/JDXI8pekBvV6Pf8kYXRZiCMYvSDsG8Cnao5/eEhyJPBQ4AtV9d9z2P7dgeO59T75QFV9t+8se+Wa637Z3w1xLHS5HA+3zbRfjoXejvyTPBbYBZwOPAF4IvASYFf3WF853rXk9gnAfwK/Bbw7ybP6ytFt/0TgMuBRwM8ABwGPBi7tHuszy2D2y5Icd0/yu0lekOT53e1D5pFlmoYyFrosg3nehzIehrRPluSY/lioql4+gJ3A5jHLjwR29pjjM0tufxw4srt9GPDZvnJ02/wicMiY5YcCX+o5y2D2S7fdE4GvAK8GTus+/rlbdmLfeab8sw1iLAzteR/KeBjSPum2O5Ox0Odpnw2MLv62t6uBA3vMsfTX6g1V9T8AVfWtJDf3mANG10ca92v+zYy/dtIsDWm/APwF8Mu116/7SQ4FLgZeP4dM0zKUsQDDet6HMh6GtE9gRmOhz/I/B/h0krdwy9VA78foshBn95jjqCTfZ/SP6U5J7l1V3+wuUXFAjzkA/hq4LMkHuWWfLACPAf6q5yxD2i8wnCKYhaGMBRjW8z6U8TCkfQIzGgu9vsire7OX32b0x5wwOvo5v6q+0FuIZXTnzx5UVZ/oebuHAo/j1vvkA1V1fZ85ljPH/XIS8JfA2CKoqnP7zDNtQx4L4HgYZ38bC3N5hW+SewA1hCdUwzXkIpgWx4JWYxZjoc/ZPgtJ3pJkN6PzVJ9KsrtbtrnHHPfrtvlfSU5NcuCSx951e187gyx/sOT2EUkuTHJ9ko8neUDPWb6T5HVJjuumIc5d9w/7ou7jQuCi/aEkhzIWuiyOh9vmaGIs9Pkir7cC7wQ2VdWWqtoCbALexegtIPtyDvBh4E+67X8kyT27x/p+U5BTltw+C3gbcE/g7xj9Zb9P1wGXAy8FrkryD0mO7TnDTyU5OsknGT1XZzLaJx9J8skkD5tXrikZylgAx8M4bYyFHqcr7VrLYzPIcfle958B7ADuD1zWV45u25fdTq7PzDHLAvCnjOZcfxX4mz6z7NkfwCPGLD+WOUy3m/LPNoixsGc/73W/+fHQyljoc7bPpUn+CTiPW89wOAn4TI85Dkxy56r6EUBV/WuSbzJ6X4KDeswBcN8k/8joHN7GJAdW1f/tydlzlp/+eltVi8DLgZcneSCjWSh9O6iqLt57YVV9Mknfz9O0DWUsgONhnCbGQp/lfyJwMqNXMu75o8XXgffQ7/S21wGPAD6yZ0FV/UeSpzJ6kvv0oiW3LwEOBq5Pcm/6f1e0i8YtrKovMnrO+vbvSS5gNId5aUGeCLx/DnmmaShjARwP4zQxFryevwYryeOBE7jtdMhJ3kBIWndmMRYGUf5JnlRV7zXHLczSpiHta7MMN8c0DOWSzr8y7wCdoeQAsywrydZ5Z5ihIe1rs9zWUHIAk42Fvl/h+wvc8qvLnsu1nl9VO3sLMaAcZlmbJM+uqtfMO8ckhrSvzTLcHCuZZCz0+SKvP2M0hznAp4BPd7ffnGRbaznMMpGfzDvAJIa0r80y3ByrtOax0NuRf5IvAQ9ZMnVrz/I7Ajtq9EKXZnKYZe2SLFbVwrxzrNWQ9rVZhptjNSYZC31O9bwZuA/wtb2Wb+oeay2HWW5Hks8t9xBweJ9ZZmBI+9osw80BzG4s9Fn+zwMuTLKLW1+Z7ue59cu6W8lhltt3OKMLWe19/ZIweoON9WxI+9osw82xx0zGQt9/8L0Dt7xv6Z65qp+uqpt6CzGgHGa53SxnA/9SVR8b89ibqurpfWeapoHta7MMNEeXZSZjYRDz/CVJ/RrKPH9JUo8sf0lqkOUvSQ2y/CWpQZa/JDXo/wGtdoDF0R5FOAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a1b7db1d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# histogram of predicted probabilities grouped by actual response value for LR\n",
    "y_test_lr_df.class_1.hist(by= y_test_lr_df.actual, sharex=True, sharey=True)\n",
    "#same for RF\n",
    "y_test_rf_df.class_1.hist(by= y_test_rf_df.actual, sharex=True, sharey=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on what threshold we choose will affect what part of each histogram consists of misclassified examples.\n",
    "\n",
    "Nonetheless, it is fruitful at this point to examine all of those test-set samples that each classifier was \"very certain\" about (had very high or very low probability) but was of the wrong category (a class 0 sample that had a very high probability or a class 1 sample with a probability very close to 0)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### ROC/Confusion Matrix/Metric Takeaways\n",
    "\n",
    "* For binary classifiers, you want to have the highest AUC possible. To decide between two classifiers, maximize the one that has the higher AUC.\n",
    "* Once you've maximized your AUC, you set about choosing your classifier threshold based on the specifics of your problem. If you need to minimize false alarms (so maximize precision) while keeping your recall some acceptable level, then you do one thing. If you need to maximize accuracy, you do something else.\n",
    "* In every case, what you're doing is generating a threshold from a specific point on the ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}