2.5 The Effect of Gun Ownership on Homocide Rates.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Effect of Gun Ownership on Homocide Rates\n",
    "\n",
    "The goal of this exercise is to attempt to measure the impact of gun ownership (logfssl) on homicide rates (logghomr) in the United States.  We will both want to determine which of these variables are useful for the prediction and then find a suitable model as well for performing the inference. \n",
    "\n",
    "## The Algorithm\n",
    "\n",
    "Here's a general outline of the method that we're using. \n",
    "\n",
    "  1.  First we will split our training data in half and label the halves the auxillary and main sample.  Our data consists of outcome (Y), treatment (X), and controls (Z). \n",
    "  2.  Using any type of model and the auxillary data, fit two predictors: Y ~ Z and X ~ Z.\n",
    "  3.  Using the main sample, obtain residuals from the two predictors.  We will label the residual for Y ~ Z as $\\hat{Y}$ and likewise the residual for X ~ Z as $\\hat{X}$. \n",
    "  4.  Fit a new OLS model $\\hat{Y} \\sim \\hat{X}$ and note the coefficient of $\\hat{X}$. \n",
    "  5.  Repeat steps 2 through 4 but reversing the roles of the auxillary and main samples.\n",
    "  6.  Average the coefficients found in steps 4 in both runs and take that as the treatment effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from tqdm import tqdm, tqdm_notebook\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import Lasso, LinearRegression\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.neural_network import MLPRegressor\n",
    "\n",
    "# An experimental Microsoft library for visualization\n",
    "from interpret import show\n",
    "from interpret.data import Marginal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\preprocessing\\data.py:625: DataConversionWarning:\n",
      "\n",
      "Data with input dtype int64, float64 were all converted to float64 by StandardScaler.\n",
      "\n",
      "D:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\base.py:462: DataConversionWarning:\n",
      "\n",
      "Data with input dtype int64, float64 were all converted to float64 by StandardScaler.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <iframe src=\"http:https://127.0.0.1:7354/2326945778992/\" width=100% height=800 frameBorder=\"0\"></iframe>\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = pd.read_csv('data/gun_clean.csv').drop(['Unnamed: 0', 'CountyCode'], axis=1)\n",
    "outcome = 'logghomr'\n",
    "treatment = 'logfssl'\n",
    "scaler = StandardScaler()\n",
    "Z = data.drop([outcome, treatment], axis=1)\n",
    "Z_c = scaler.fit_transform(Z)\n",
    "X = data[[treatment]]\n",
    "X_c = scaler.fit_transform(X).reshape(-1)\n",
    "y = data[outcome]\n",
    "marginals = Marginal().explain_data(X, y, name=\"Gun Data\")\n",
    "# This will enable us to view the various marginal distributions between any variable and the outcome.\n",
    "# However with nearly 500 variables, it's impractical to go through each one at a time.  This motivates why \n",
    "# we might use both dimensionality reduction and variable selection methods. \n",
    "show(marginals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modeling with Lasso\n",
    "\n",
    "Lasso will serve as our baseline model, and then we can have a little more fun. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "def measure_treatment_effect(X, Z, y, model, model_args={}):\n",
    "    X_main, X_aux, Z_main, Z_aux, y_main, y_aux = train_test_split(X, Z, y, test_size=0.5)\n",
    "    #print(X_main.shape, Z_main.shape, y_main.shape)\n",
    "    # Step 1\n",
    "    model_X_1 = model(**model_args).fit(X=Z_aux, y=X_aux)\n",
    "    resid_X_1 = X_main - model_X_1.predict(Z_main)\n",
    "    model_y_1 = model(**model_args).fit(X=Z_aux, y=y_aux)\n",
    "    resid_y_1 = y_main - model_y_1.predict(Z_main)\n",
    "    effect_1 = LinearRegression().fit(X=resid_X_1.reshape(-1, 1), y=resid_y_1).coef_\n",
    "    # Step 2\n",
    "    model_X_2 = model(**model_args).fit(X=Z_main, y=X_main)\n",
    "    resid_X_2 = X_aux - model_X_2.predict(Z_aux)\n",
    "    model_y_2 = model(**model_args).fit(X=Z_main, y=y_main)\n",
    "    resid_y_2 = y_aux - model_y_2.predict(Z_aux)\n",
    "    effect_2 = LinearRegression().fit(X=resid_X_2.reshape(-1, 1), y=resid_y_2).coef_\n",
    "    # Step 3\n",
    "    return (effect_1 + effect_2) / 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████| 100/100 [00:16<00:00,  6.42it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x21dc4945c88>"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAD8dJREFUeJzt3X+MHPV5x/H3AwaV+CiQOpwcF+WgolEpl5D4RFPRVndCpASrAf6IGkIRVlI5kSCNKquNm7Zq1CiSFYWkrZo/agkIlRpO/QEKAgqliAXR0jR3yGA7Vsqva2qwsFzAYR2k9Nynf+wYtpcztz9mb/e+fr+k083Mznz3eW5vPpqbndmLzESSVI5Thl2AJKleBrskFcZgl6TCGOySVBiDXZIKY7BLUmEMdkkqjMEuSYUx2CWpMOtW88k2bNiQExMTXW939OhR1q9fX39BQ2I/o81+RtvJ2M/8/PzhzHxXp2OuarBPTEwwNzfX9XaNRoPp6en6CxoS+xlt9jPaTsZ+IuI/uxnTUzGSVBiDXZIKY7BLUmEMdkkqjMEuSYVZMdgj4ryIeCQi9kfEvoj4XLX8ixHxYkTsrr6uGny5kqSVdHK54yKwPTOfjIgzgfmIeKh67OuZ+dXBlSdJ6taKwZ6ZB4GD1fTrEbEf2DTowiRJvenqHHtETAAfAL5TLbo5Ip6OiNsi4pyaa5Mk9SA6/WfWETEGPAp8OTPviohx4DCQwJeAjZn5yWW22wZsAxgfH988OzvbdZHNZpMXjhzreru6TG46q9bxms0mY2NjtY45TPYz2uxntHXSz8zMzHxmTnU6ZkfBHhGnAfcCD2bm15Z5fAK4NzMvfrtxpqamstePFNj6wNGut6vLws4ttY53Mt4SvZbYz2g7GfuJiK6CvZOrYgK4FdjfHuoRsbFttWuBvZ0+qSRpcDq5KuYy4AZgT0TsrpZ9AbguIi6hdSpmAfj0QCqUJHWlk6tiHgdimYfur78cSVK/vPNUkgpjsEtSYQx2SSqMwS5JhTHYJakwBrskFcZgl6TCGOySVBiDXZIKY7BLUmEMdkkqjMEuSYUx2CWpMAa7JBXGYJekwhjsklQYg12SCmOwS1JhDHZJKozBLkmFMdglqTAGuyQVxmCXpMIY7JJUGINdkgpjsEtSYQx2SSqMwS5JhTHYJakwBrskFcZgl6TCGOySVJgVgz0izouIRyJif0Tsi4jPVcvfGREPRcQz1fdzBl+uJGklnRyxLwLbM/MXgA8BN0XERcAO4OHMvBB4uJqXJA3ZisGemQcz88lq+nVgP7AJuBq4o1rtDuCaQRUpSepcV+fYI2IC+ADwHWA8Mw9CK/yBc+suTpLUvcjMzlaMGAMeBb6cmXdFxGuZeXbb469m5k+cZ4+IbcA2gPHx8c2zs7NdF9lsNnnhyLGut6vL5Kazah2v2WwyNjZW65jDZD+jzX5GWyf9zMzMzGfmVKdjrutkpYg4DfgH4G8y865q8csRsTEzD0bERuDQcttm5i5gF8DU1FROT093WtubGo0Gtzx+tOvt6rJw/XSt4zUaDXr5OYwq+xlt9jPaBtFPJ1fFBHArsD8zv9b20D3AjdX0jcC3a61MktSTTo7YLwNuAPZExO5q2ReAncDfRsSngB8AHxtMiZKkbqwY7Jn5OBAnePjyesuRJPXLO08lqTAGuyQVxmCXpMIY7JJUGINdkgpjsEtSYQx2SSqMwS5JhTHYJakwBrskFcZgl6TCGOySVJiOPo9dJ5eJHfd1tf72yUW2drnNiSzs3FLLONLJzCN2SSqMwS5JhTHYJakwBrskFcZgl6TCGOySVBiDXZIKY7BLUmEMdkkqjMEuSYUx2CWpMAa7JBXGYJekwhjsklQYg12SCmOwS1JhDHZJKozBLkmFMdglqTAGuyQVZsVgj4jbIuJQROxtW/bFiHgxInZXX1cNtkxJUqc6OWL/JnDlMsu/npmXVF/311uWJKlXKwZ7Zj4GvLIKtUiSatDPOfabI+Lp6lTNObVVJEnqS2TmyitFTAD3ZubF1fw4cBhI4EvAxsz85Am23QZsAxgfH988OzvbdZHNZpMXjhzreru6TG46q9bxms0mY2NjtY5Zpz0vHulq/fEz4OU36nnuun/WvRj116db9jPaOulnZmZmPjOnOh2zp2Dv9LGlpqamcm5urtPa3tRoNNj6wNGut6vLws4ttY7XaDSYnp6udcw6Tey4r6v1t08ucsuedbU8d90/616M+uvTLfsZbZ30ExFdBXtPp2IiYmPb7LXA3hOtK0laXSseZkXEncA0sCEiDgB/AkxHxCW0TsUsAJ8eYI2SpC6sGOyZed0yi28dQC2SpBp456kkFcZgl6TCGOySVJh6rlHTQHR72aEkgUfsklQcg12SCmOwS1JhDHZJKozBLkmFMdglqTAGuyQVxmCXpMIY7JJUGINdkgpjsEtSYQx2SSqMwS5JhTHYJakwBrskFcbPY9dIGdZn0C/s3DKU55UGwSN2SSqMwS5JhTHYJakwBrskFcZgl6TCGOySVBiDXZIKY7BLUmG8QakDdd80s31yka1DuhFHUvk8YpekwhjsklQYg12SCrNisEfEbRFxKCL2ti17Z0Q8FBHPVN/PGWyZkqROdXLE/k3gyiXLdgAPZ+aFwMPVvCRpBKwY7Jn5GPDKksVXA3dU03cA19RclySpR72eYx/PzIMA1fdz6ytJktSPyMyVV4qYAO7NzIur+dcy8+y2x1/NzGXPs0fENmAbwPj4+ObZ2dmui2w2m7xw5FjX242q8TPg5TeGXUV9SuhnctNZb043m03GxsaGWE297Ge0ddLPzMzMfGZOdTpmrzcovRwRGzPzYERsBA6daMXM3AXsApiamsrp6emun6zRaHDL40d7LHX0bJ9c5JY95dwbVkI/C9dPvzndaDTo5fd0VNnPaBtEP72eirkHuLGavhH4dj3lSJL61cnljncCTwDvjYgDEfEpYCdwRUQ8A1xRzUuSRsCKfz9n5nUneOjymmuRJNXAO08lqTAGuyQVxmCXpMIY7JJUGINdkgpjsEtSYQx2SSqMwS5JhTHYJakwBrskFcZgl6TCGOySVBiDXZIKY7BLUmEMdkkqjMEuSYUx2CWpMAa7JBXGYJekwhjsklQYg12SCmOwS1JhDHZJKozBLkmFMdglqTAGuyQVxmCXpMIY7JJUmHXDLkAaBRM77ntzevvkIlvb5gdtYeeWVXsunRw8YpekwhjsklQYg12SCmOwS1Jh+nrzNCIWgNeBY8BiZk7VUZQkqXd1XBUzk5mHaxhHklQDT8VIUmEiM3vfOOIF4FUggb/KzF3LrLMN2AYwPj6+eXZ2tuvnaTabvHDkWM91jprxM+DlN4ZdRX3sZ7SdqJ/JTWetfjE1aDabjI2NDbuM2nTSz8zMzHw3p7r7DfZ3Z+ZLEXEu8BDw2cx87ETrT01N5dzcXNfP02g02PrA0Z7rHDXbJxe5ZU8594bZz2g7UT9r9caoRqPB9PT0sMuoTSf9RERXwd7XqZjMfKn6fgi4G7i0n/EkSf3rOdgjYn1EnHl8GvgwsLeuwiRJvenn781x4O6IOD7OtzLzgVqqkiT1rOdgz8zngffXWIskqQZe7ihJhTHYJakwBrskFaaci3UldWViFf+ZyFJr9Rr6tcIjdkkqjMEuSYUx2CWpMAa7JBXGYJekwhjsklQYg12SCuN17JJOGifLtfsesUtSYQx2SSqMwS5JhTHYJakwBrskFcZgl6TCGOySVBivY5e06vq5nnz75CJbh3g9+lrgEbskFcZgl6TCGOySVBiDXZIKY7BLUmEMdkkqjMEuSYUx2CWpMAa7JBXGYJekwhjsklQYg12SCtNXsEfElRHx/Yh4NiJ21FWUJKl3PQd7RJwKfAP4CHARcF1EXFRXYZKk3vRzxH4p8GxmPp+ZPwZmgavrKUuS1Kt+gn0T8F9t8weqZZKkIYrM7G3DiI8Bv56Zv13N3wBcmpmfXbLeNmBbNfte4Ps9PN0G4HBPhY4m+xlt9jPaTsZ+3pOZ7+p0wH7+g9IB4Ly2+Z8FXlq6UmbuAnb18TxExFxmTvUzxiixn9FmP6PNflbWz6mY7wIXRsT5EXE68HHgnnrKkiT1qucj9sxcjIibgQeBU4HbMnNfbZVJknrS1z+zzsz7gftrquXt9HUqZwTZz2izn9FmPyvo+c1TSdJo8iMFJKkwQwn2lT6KICI+ExF7ImJ3RDzefkdrRLwvIp6IiH3VOj9VLW9UY+6uvs4d9X4i4vq2endHxP9GxCXVY5urbZ6NiL+IiFjDvQzttemzp9Mi4o7qsf0R8QedjrnGello22ZutXrps5/TI+L26rGnImK6bZuh7DsD7Kf7/SczV/WL1hutzwEXAKcDTwEXLVnnp9umPwo8UE2vA54G3l/N/wxwajXdAKbWUj9L1pkEnm+b/3fgl4EA/hH4yBruZSivTQ2/b58AZqvpdwALwEQnY66VXqr5BWDDGnttbgJur6bPBeaBU6r5Vd93BtxP1/vPMI7YV/wogsz8YdvseuD4GwEfBp7OzKeq9f47M4+tQs1vp59+2l0H3AkQERtp/QI8ka1X9q+BawZR/BK19zIC+ukpgfURsQ44A/gx8MNOxhyQQfQyTP30cxHwcLXOIeA1YGqI+w4MoJ9eCxlGsHf0UQQRcVNEPAd8BfidavHPAxkRD0bEkxHx+0s2u736U+WPV/HPr376afebvBWGm6px3nbMARhEL8cN47WB/nr6e+AocBD4AfDVzHyl0zEHYBC9QCtc/iki5qN1p/hq6aefp4CrI2JdRJwPbKZ1w+Sw9h0YTD/HdbX/DCPYlyvqJ476MvMbmflzwOeBP6oWrwN+Bbi++n5tRFxePXZ9Zk4Cv1p93VB34SfQTz+tASJ+CfhRZu7tZswBGEQvMLzXBvrr6VLgGPBu4Hxge0Rc0OmYAzCIXgAuy8wP0vqk1psi4tdqr3x5/fRzG63gnAP+DPhXYLHTMQdkEP1AD/vPMIK9o48iaDPLW39KHQAezczDmfkjWtfQfxAgM1+svr8OfIvWL/Jq6Kef4z7O/z/CPVCN0+mYdRlEL8N8baC/nj5B6xzo/1R/Hv8LrT+Pux2zLoPohcx8qfp+CLibNbDvZOZiZv5uZl6SmVcDZwPPMLx9BwbTT2/7z2q8qbDkzYN1wPO0jhqOv8Hwi0vWubBt+jeAuWr6HOBJWm/+rAP+GdhSTW+o1jmN1p+dnxn1fqr5U6pfiAuWbPNd4EO89QbQVWuxl2G+NjX8vn0euL16DdYD3wPe18mYa6iX9cCZ1TrraR0pXrkGXpt3AOur6SuAx9rWW/V9Z1D99Lr/rMrOtcwP4CrgP2i9g/yH1bI/BT5aTf85sA/YDTzS/sMBfqt6bC/wlbZfyHlaV8zsq7Y/dY30Mw382zJjTlU9Pgf8JdXNZGutl2G/Nv30BIwBf1c99j3g995uzLXYC60rOJ6qvvatZi999jNB65Ni99M6wHvPsPedQfTT6/7jnaeSVBjvPJWkwhjsklQYg12SCmOwS1JhDHZJKozBLkmFMdglqTAGuyQV5v8AbHCT87jry9YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bootstraps = 100\n",
    "ates = pd.DataFrame([measure_treatment_effect(X_c, Z_c, y, Lasso, {'alpha': 1.}) for i in tqdm(range(bootstraps))], columns=['ATE'])\n",
    "ates['ATE'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  0%|                                                  | 0/100 [00:00<?, ?it/s]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  1%|▍                                         | 1/100 [00:04<07:40,  4.65s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  2%|▊                                         | 2/100 [00:09<07:35,  4.64s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  3%|█▎                                        | 3/100 [00:13<07:29,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  4%|█▋                                        | 4/100 [00:18<07:24,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  5%|██                                        | 5/100 [00:23<07:21,  4.65s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  6%|██▌                                       | 6/100 [00:27<07:16,  4.65s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  7%|██▉                                       | 7/100 [00:32<07:12,  4.65s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  8%|███▎                                      | 8/100 [00:37<07:07,  4.64s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  9%|███▊                                      | 9/100 [00:41<07:02,  4.64s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 10%|████                                     | 10/100 [00:46<06:56,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 11%|████▌                                    | 11/100 [00:50<06:51,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 12%|████▉                                    | 12/100 [00:55<06:50,  4.66s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 13%|█████▎                                   | 13/100 [01:00<06:44,  4.64s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 14%|█████▋                                   | 14/100 [01:04<06:37,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 15%|██████▏                                  | 15/100 [01:09<06:31,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 16%|██████▌                                  | 16/100 [01:14<06:25,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 17%|██████▉                                  | 17/100 [01:18<06:21,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 18%|███████▍                                 | 18/100 [01:23<06:16,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 19%|███████▊                                 | 19/100 [01:27<06:11,  4.58s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 20%|████████▏                                | 20/100 [01:32<06:07,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 21%|████████▌                                | 21/100 [01:37<06:03,  4.61s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 22%|█████████                                | 22/100 [01:41<05:59,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 23%|█████████▍                               | 23/100 [01:46<05:54,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 24%|█████████▊                               | 24/100 [01:50<05:49,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 25%|██████████▎                              | 25/100 [01:55<05:46,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 26%|██████████▋                              | 26/100 [02:00<05:43,  4.65s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 27%|███████████                              | 27/100 [02:04<05:37,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 28%|███████████▍                             | 28/100 [02:09<05:33,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 29%|███████████▉                             | 29/100 [02:13<05:27,  4.61s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 30%|████████████▎                            | 30/100 [02:18<05:23,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 31%|████████████▋                            | 31/100 [02:23<05:19,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 32%|█████████████                            | 32/100 [02:27<05:13,  4.61s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 33%|█████████████▌                           | 33/100 [02:32<05:08,  4.61s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 34%|█████████████▉                           | 34/100 [02:36<05:02,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 35%|██████████████▎                          | 35/100 [02:41<04:57,  4.58s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 36%|██████████████▊                          | 36/100 [02:46<04:54,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 37%|███████████████▏                         | 37/100 [02:50<04:49,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 38%|███████████████▌                         | 38/100 [02:55<04:45,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 39%|███████████████▉                         | 39/100 [02:59<04:40,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 40%|████████████████▍                        | 40/100 [03:04<04:37,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 41%|████████████████▊                        | 41/100 [03:09<04:32,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 42%|█████████████████▏                       | 42/100 [03:13<04:28,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 43%|█████████████████▋                       | 43/100 [03:18<04:22,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 44%|██████████████████                       | 44/100 [03:23<04:17,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 45%|██████████████████▍                      | 45/100 [03:27<04:12,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 46%|██████████████████▊                      | 46/100 [03:32<04:07,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 47%|███████████████████▎                     | 47/100 [03:36<04:03,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 48%|███████████████████▋                     | 48/100 [03:41<03:58,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 49%|████████████████████                     | 49/100 [03:46<03:54,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 50%|████████████████████▌                    | 50/100 [03:50<03:50,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 51%|████████████████████▉                    | 51/100 [03:55<03:45,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 52%|█████████████████████▎                   | 52/100 [03:59<03:40,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 53%|█████████████████████▋                   | 53/100 [04:04<03:35,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 54%|██████████████████████▏                  | 54/100 [04:09<03:32,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 55%|██████████████████████▌                  | 55/100 [04:13<03:28,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 56%|██████████████████████▉                  | 56/100 [04:18<03:22,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 57%|███████████████████████▎                 | 57/100 [04:22<03:17,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 58%|███████████████████████▊                 | 58/100 [04:27<03:12,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 59%|████████████████████████▏                | 59/100 [04:31<03:08,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 60%|████████████████████████▌                | 60/100 [04:36<03:03,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 61%|█████████████████████████                | 61/100 [04:41<02:59,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 62%|█████████████████████████▍               | 62/100 [04:45<02:54,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 63%|█████████████████████████▊               | 63/100 [04:50<02:49,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 64%|██████████████████████████▏              | 64/100 [04:54<02:44,  4.58s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 65%|██████████████████████████▋              | 65/100 [04:59<02:40,  4.58s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 66%|███████████████████████████              | 66/100 [05:04<02:35,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 67%|███████████████████████████▍             | 67/100 [05:08<02:31,  4.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 68%|███████████████████████████▉             | 68/100 [05:13<02:27,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 69%|████████████████████████████▎            | 69/100 [05:18<02:23,  4.64s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 70%|████████████████████████████▋            | 70/100 [05:22<02:19,  4.65s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 71%|█████████████████████████████            | 71/100 [05:27<02:14,  4.64s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 72%|█████████████████████████████▌           | 72/100 [05:32<02:10,  4.68s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 73%|█████████████████████████████▉           | 73/100 [05:36<02:06,  4.67s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 74%|██████████████████████████████▎          | 74/100 [05:41<02:00,  4.64s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 75%|██████████████████████████████▊          | 75/100 [05:45<01:55,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 76%|███████████████████████████████▏         | 76/100 [05:50<01:50,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 77%|███████████████████████████████▌         | 77/100 [05:55<01:46,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 78%|███████████████████████████████▉         | 78/100 [05:59<01:41,  4.61s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 79%|████████████████████████████████▍        | 79/100 [06:04<01:36,  4.61s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 80%|████████████████████████████████▊        | 80/100 [06:09<01:32,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 81%|█████████████████████████████████▏       | 81/100 [06:13<01:28,  4.66s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 82%|█████████████████████████████████▌       | 82/100 [06:18<01:23,  4.66s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 83%|██████████████████████████████████       | 83/100 [06:23<01:19,  4.69s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 84%|██████████████████████████████████▍      | 84/100 [06:27<01:14,  4.66s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 85%|██████████████████████████████████▊      | 85/100 [06:32<01:09,  4.66s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 86%|███████████████████████████████████▎     | 86/100 [06:37<01:04,  4.64s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 87%|███████████████████████████████████▋     | 87/100 [06:41<01:00,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 88%|████████████████████████████████████     | 88/100 [06:46<00:55,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 89%|████████████████████████████████████▍    | 89/100 [06:51<00:51,  4.69s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 90%|████████████████████████████████████▉    | 90/100 [06:55<00:46,  4.66s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 91%|█████████████████████████████████████▎   | 91/100 [07:00<00:41,  4.65s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 92%|█████████████████████████████████████▋   | 92/100 [07:04<00:36,  4.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 93%|██████████████████████████████████████▏  | 93/100 [07:09<00:32,  4.61s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 94%|██████████████████████████████████████▌  | 94/100 [07:14<00:27,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 95%|██████████████████████████████████████▉  | 95/100 [07:18<00:22,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 96%|███████████████████████████████████████▎ | 96/100 [07:23<00:18,  4.60s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 97%|███████████████████████████████████████▊ | 97/100 [07:27<00:13,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 98%|████████████████████████████████████████▏| 98/100 [07:32<00:09,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 99%|████████████████████████████████████████▌| 99/100 [07:37<00:04,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "100%|████████████████████████████████████████| 100/100 [07:41<00:00,  4.63s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x21dc4aaa400>"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADhJJREFUeJzt3W+MHIddxvHnIaFp8IHr4GRlTNtLJBPh9KCRjwqBgD2FkrRWmyBaqYFWDg06/qQFCb8xBIkKhGQhAuJFpCpSo5gX9FpQS6MktHVNrlElWjinJucQGSeugTiWo/yp6QWr6MqPFzcXra8+7+zM7Ozuj+9HWt3s7Px5brz3eG7mZscRIQDA5PueUQcAADSDQgeAJCh0AEiCQgeAJCh0AEiCQgeAJCh0AEiCQgeAJCh0AEjiyjZXtn379pienm5zlRd57bXXtGXLlpGtvypyt2cSM0vkblvbuY8ePfpSRFzbb7pWC316elpLS0ttrvIii4uL6na7I1t/VeRuzyRmlsjdtrZz2/73MtNxyAUAkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0Akmj1SlFMhukDjzaynP0zq7qroWUN2+mDe0cdAaiNPXQASIJCB4AkKHQASIJCB4AkKHQASIJCB4AkKHQASIJCB4AkKHQASIJCB4AkKHQASIJCB4AkKHQASIJCB4Ak+ha67Tfbftz2M7aftv07xfhrbB+2fbL4um34cQEAmymzh74qaX9E/Kikn5R0j+3dkg5IOhIRuyQdKZ4DAEakb6FHxNmIeLIY/pakZyTtlHS7pEPFZIck3TGskACA/gY6hm57WtLNkr4mqRMRZ6W10pd0XdPhAADlOSLKTWhPSfqypD+JiM/Y/mZEvKnn9Vcj4ruOo9uelzQvSZ1OZ8/CwkIzyStYWVnR1NTUyNZfVdu5l8+cb2Q5naulcxcaWdTQzezcKon3SNvIXc7c3NzRiJjtN12pQrf9vZIekfSFiPjzYtwJSd2IOGt7h6TFiLjxcsuZnZ2NpaWlUt/AMCwuLqrb7Y5s/VW1nbvJe4retzwZt61dv6co75F2kbsc26UKvcxfuVjSJyQ9s17mhYcl7SuG90n6XJWgAIBmlNl9+mlJH5K0bPtYMe73JR2U9Gnbd0v6D0nvH05EAEAZfQs9Ir4iyZu8fEuzcQAAVXGlKAAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkMRl3HwCGbP2mHvtnVnVXQzf4KGv95hpAXeyhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJNG30G0/aPtF28d7xn3M9hnbx4rHu4cbEwDQT5k99Ick3XaJ8X8REW8vHo81GwsAMKi+hR4RT0h6pYUsAIAa6hxD/4jtp4pDMtsaSwQAqMQR0X8ie1rSIxHxtuJ5R9JLkkLSH0vaEREf3mTeeUnzktTpdPYsLCw0EryKlZUVTU1NjWz9VbWde/nM+UaW07laOnehkUW1ZhSZZ3Zurb0M3tvtajv33Nzc0YiY7TddpUIv+9pGs7OzsbS01Hd9w7K4uKhutzuy9VfVdu7pA482spz9M6u6b/nKRpbVllFkPn1wb+1l8N5uV9u5bZcq9EqHXGzv6Hn6i5KObzYtAKAdfXdFbH9SUlfSdtvPS/pDSV3bb9faIZfTkn59iBkBACX0LfSIuPMSoz8xhCwAgBq4UhQAkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASAJCh0AkqDQASCJvoVu+0HbL9o+3jPuGtuHbZ8svm4bbkwAQD9l9tAfknTbhnEHJB2JiF2SjhTPAQAj1LfQI+IJSa9sGH27pEPF8CFJdzScCwAwoKrH0DsRcVaSiq/XNRcJAFCFI6L/RPa0pEci4m3F829GxJt6Xn81Ii55HN32vKR5Sep0OnsWFhYaiF3NysqKpqamRrb+qtrOvXzmfCPL6VwtnbvQyKJaM4rMMzu31l4G7+12tZ17bm7uaETM9pvuyorLP2d7R0Sctb1D0oubTRgRD0h6QJJmZ2ej2+1WXGV9i4uLGuX6q2o7910HHm1kOftnVnXfctW32GiMIvPpX+nWXgbv7XaNa+6qh1welrSvGN4n6XPNxAEAVFXmzxY/KekfJd1o+3nbd0s6KOmdtk9KemfxHAAwQn1/t4yIOzd56ZaGswAAauBKUQBIgkIHgCQodABIgkIHgCQm64+EATRmuqHrDao4fXDvyNadGXvoAJAEhQ4ASVDoAJAEhQ4ASVDoAJAEhQ4ASVDoAJAEhQ4ASXBh0Rhbv/Bj/8xqYzedAJAXe+gAkASFDgBJUOgAkASFDgBJUOgAkASFDgBJUOgAkASFDgBJUOgAkARXigIj1sSt4LiaGBJ76ACQBoUOAElQ6ACQBIUOAElQ6ACQBIUOAElQ6ACQBIUOAElQ6ACQBIUOAElQ6ACQBIUOAElQ6ACQBIUOAEnU+vhc26clfUvSdyStRsRsE6EAAINr4vPQ5yLipQaWAwCogUMuAJBE3UIPSV+0fdT2fBOBAADVOCKqz2z/UES8YPs6SYclfTQintgwzbykeUnqdDp7FhYWKq1r+cz5yjnXda6Wzl0YfL6ZnVtrr7uK9e+5au5Rm8Tck5hZIndZTf0sr6ysaGpqqpFllTE3N3e0zDnKWoV+0YLsj0laiYg/22ya2dnZWFpaqrT8pu67eN/y4KcNTh/cW3vdVax/z1Vzj9ok5p7EzBK5y2rqZ3lxcVHdbreRZZVhu1ShVz7kYnuL7e9fH5b0C5KOV10eAKCeOv81diR91vb6cv46Ij7fSCoAwMAqF3pEnJL04w1mAQDUwJ8tAkASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJEGhA0ASFDoAJDF5tzgZgSbulgQAw8YeOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkQaEDQBIUOgAkwR2LAPy/0dTdx/bPrOquAZd1+uDeRtZ9OeyhA0ASFDoAJEGhA0ASFDoAJEGhA0AStQrd9m22T9h+1vaBpkIBAAZXudBtXyHpfknvkrRb0p22dzcVDAAwmDp76O+Q9GxEnIqI/5G0IOn2ZmIBAAZVp9B3SvrPnufPF+MAACPgiKg2o/1+SbdGxK8Vzz8k6R0R8dEN081Lmi+e3ijpRPW4tW2X9NII118VudsziZklcret7dxvjYhr+01U59L/5yW9uef5D0t6YeNEEfGApAdqrKcxtpciYnbUOQZF7vZMYmaJ3G0b19x1Drn8s6Rdtq+3/QZJH5D0cDOxAACDqryHHhGrtj8i6QuSrpD0YEQ83VgyAMBAan3aYkQ8JumxhrK0YSwO/VRA7vZMYmaJ3G0by9yVT4oCAMYLl/4DQBIpCr3fRxDYvsr2p4rXv2Z7uue13yvGn7B96yTktj1t+4LtY8Xj42OW+2dtP2l71fb7Nry2z/bJ4rGvvdS1c3+nZ3u3evK/RO7ftf2vtp+yfcT2W3teG+ftfbnc47y9f8P2cpHtK71XyI+yTyRJETHRD62dkH1O0g2S3iDpXyTt3jDNb0n6eDH8AUmfKoZ3F9NfJen6YjlXTEDuaUnHx3h7T0v6MUl/Jel9PeOvkXSq+LqtGN427rmL11bGeHvPSfq+Yvg3e94n4769L5l7Arb3D/QMv1fS54vhkfXJ+iPDHnqZjyC4XdKhYvhvJd1i28X4hYj4dkR8Q9KzxfLGPfco9c0dEacj4ilJ/7th3lslHY6IVyLiVUmHJd3WRmjVyz1KZXI/HhH/XTz9qtauCZHGf3tvlnuUyuT+r56nWyStn4gcZZ9IynHIpcxHELw+TUSsSjov6QdLzjssdXJL0vW2v277y7Z/ZthhL5WpMMg2G/ftfTlvtL1k+6u272g22mUNmvtuSX9fcd4m1cktjfn2tn2P7eck/amk3x5k3mHKcJPoS+2xbvzTnc2mKTPvsNTJfVbSWyLiZdt7JP2d7Zs27DkMS51tNu7b+3LeEhEv2L5B0j/YXo6I5xrKdjmlc9v+oKRZST836LxDUCe3NObbOyLul3S/7V+W9AeS9pWdd5gy7KGX+QiC16exfaWkrZJeKTnvsFTOXfxK97IkRcRRrR2r+5GhJ96QqTDINhv37b2piHih+HpK0qKkm5sMdxmlctv+eUn3SnpvRHx7kHmHpE7usd/ePRYkrf8GMcrtvWYUJx6afGjtt4xTWjsJsX4S46YN09yji08ufroYvkkXn8Q4pfZOitbJfe16Tq2dvDkj6Zpxyd0z7UP67pOi39DaCbptxfAk5N4m6apieLukk9pwomzE75Obtfaf+q4N48d6e18m97hv7109w++RtFQMj6xPXs/T5sqG+I/wbkn/Vrw57i3G/ZHW/teXpDdK+hutnaT4J0k39Mx7bzHfCUnvmoTckn5J0tPFm+dJSe8Zs9w/obW9ldckvSzp6Z55P1x8P89K+tVJyC3ppyQtF9t7WdLdY5b7S5LOSTpWPB6ekO19ydwTsL3/svj5OybpcfUU/ij7JCK4UhQAsshwDB0AIAodANKg0AEgCQodAJKg0AEgCQodAJKg0AEgCQodAJL4P1VtHV4agDCsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bootstraps = 100\n",
    "ates = pd.DataFrame([measure_treatment_effect(X_c, Z_c, y, RandomForestRegressor, {'n_estimators': 10}) for i in tqdm(range(bootstraps))], columns=['ATE'])\n",
    "ates['ATE'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  0%|                                                  | 0/100 [00:00<?, ?it/s]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  1%|▍                                         | 1/100 [00:09<15:13,  9.23s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  2%|▊                                         | 2/100 [00:16<14:10,  8.68s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  3%|█▎                                        | 3/100 [00:23<13:10,  8.15s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  4%|█▋                                        | 4/100 [00:30<12:14,  7.66s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  5%|██                                        | 5/100 [00:37<12:06,  7.65s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  6%|██▌                                       | 6/100 [00:45<11:53,  7.59s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  7%|██▉                                       | 7/100 [00:54<12:25,  8.01s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  8%|███▎                                      | 8/100 [01:03<12:54,  8.42s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  9%|███▊                                      | 9/100 [01:13<13:16,  8.75s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 10%|████                                     | 10/100 [01:20<12:41,  8.47s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 11%|████▌                                    | 11/100 [01:28<12:04,  8.14s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 12%|████▉                                    | 12/100 [01:37<12:24,  8.46s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 13%|█████▎                                   | 13/100 [01:44<11:28,  7.92s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 14%|█████▋                                   | 14/100 [01:53<11:54,  8.30s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 15%|██████▏                                  | 15/100 [02:00<11:08,  7.86s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 16%|██████▌                                  | 16/100 [02:07<10:56,  7.82s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 17%|██████▉                                  | 17/100 [02:16<11:16,  8.15s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 18%|███████▍                                 | 18/100 [02:24<10:47,  7.90s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 19%|███████▊                                 | 19/100 [02:32<10:50,  8.03s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 20%|████████▏                                | 20/100 [02:38<10:00,  7.50s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 21%|████████▌                                | 21/100 [02:45<09:35,  7.29s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 22%|█████████                                | 22/100 [02:54<10:11,  7.84s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 23%|█████████▍                               | 23/100 [03:02<09:57,  7.76s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 24%|█████████▊                               | 24/100 [03:11<10:15,  8.10s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 25%|██████████▎                              | 25/100 [03:17<09:40,  7.73s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 26%|██████████▋                              | 26/100 [03:24<08:59,  7.29s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 27%|███████████                              | 27/100 [03:32<09:12,  7.57s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 28%|███████████▍                             | 28/100 [03:39<08:55,  7.44s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 29%|███████████▉                             | 29/100 [03:46<08:46,  7.42s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 30%|████████████▎                            | 30/100 [03:54<08:39,  7.42s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 31%|████████████▋                            | 31/100 [04:03<09:07,  7.94s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 32%|█████████████                            | 32/100 [04:11<08:55,  7.88s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 33%|█████████████▌                           | 33/100 [04:17<08:20,  7.47s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 34%|█████████████▉                           | 34/100 [04:24<07:59,  7.26s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 35%|██████████████▎                          | 35/100 [04:30<07:36,  7.02s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 36%|██████████████▊                          | 36/100 [04:37<07:20,  6.88s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 37%|███████████████▏                         | 37/100 [04:46<07:44,  7.37s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 38%|███████████████▌                         | 38/100 [04:53<07:44,  7.48s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 39%|███████████████▉                         | 39/100 [05:03<08:24,  8.27s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 40%|████████████████▍                        | 40/100 [05:11<08:04,  8.07s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 41%|████████████████▊                        | 41/100 [05:20<08:17,  8.44s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 42%|█████████████████▏                       | 42/100 [05:28<07:54,  8.18s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 43%|█████████████████▋                       | 43/100 [05:37<08:01,  8.45s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 44%|██████████████████                       | 44/100 [05:46<08:09,  8.75s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 45%|██████████████████▍                      | 45/100 [05:53<07:33,  8.25s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 46%|██████████████████▊                      | 46/100 [06:00<06:51,  7.62s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 47%|███████████████████▎                     | 47/100 [06:09<07:07,  8.07s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 48%|███████████████████▋                     | 48/100 [06:15<06:31,  7.52s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 49%|████████████████████                     | 49/100 [06:23<06:25,  7.56s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 50%|████████████████████▌                    | 50/100 [06:29<06:02,  7.26s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 51%|████████████████████▉                    | 51/100 [06:38<06:13,  7.61s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 52%|█████████████████████▎                   | 52/100 [06:46<06:13,  7.78s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 53%|█████████████████████▋                   | 53/100 [06:54<06:06,  7.80s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 54%|██████████████████████▏                  | 54/100 [07:03<06:25,  8.39s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 55%|██████████████████████▌                  | 55/100 [07:12<06:14,  8.31s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 56%|██████████████████████▉                  | 56/100 [07:18<05:45,  7.85s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 57%|███████████████████████▎                 | 57/100 [07:26<05:40,  7.92s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 58%|███████████████████████▊                 | 58/100 [07:35<05:36,  8.02s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 59%|████████████████████████▏                | 59/100 [07:42<05:24,  7.91s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 60%|████████████████████████▌                | 60/100 [07:50<05:17,  7.95s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 61%|█████████████████████████                | 61/100 [07:58<05:04,  7.81s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 62%|█████████████████████████▍               | 62/100 [08:06<05:06,  8.06s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 63%|█████████████████████████▊               | 63/100 [08:14<04:52,  7.90s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 64%|██████████████████████████▏              | 64/100 [08:25<05:17,  8.81s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 65%|██████████████████████████▋              | 65/100 [08:32<04:49,  8.27s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 66%|███████████████████████████              | 66/100 [08:40<04:41,  8.28s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 67%|███████████████████████████▍             | 67/100 [08:48<04:30,  8.19s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 68%|███████████████████████████▉             | 68/100 [08:56<04:16,  8.03s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 69%|████████████████████████████▎            | 69/100 [09:03<04:02,  7.82s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 70%|████████████████████████████▋            | 70/100 [09:13<04:10,  8.35s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 71%|█████████████████████████████            | 71/100 [09:20<03:55,  8.12s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 72%|█████████████████████████████▌           | 72/100 [09:29<03:49,  8.20s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 73%|█████████████████████████████▉           | 73/100 [09:36<03:30,  7.78s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 74%|██████████████████████████████▎          | 74/100 [09:45<03:33,  8.20s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 75%|██████████████████████████████▊          | 75/100 [09:53<03:24,  8.19s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 76%|███████████████████████████████▏         | 76/100 [10:01<03:15,  8.15s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 77%|███████████████████████████████▌         | 77/100 [10:08<02:58,  7.75s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 78%|███████████████████████████████▉         | 78/100 [10:16<02:56,  8.01s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 79%|████████████████████████████████▍        | 79/100 [10:25<02:50,  8.10s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 80%|████████████████████████████████▊        | 80/100 [10:31<02:31,  7.57s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 81%|█████████████████████████████████▏       | 81/100 [10:38<02:21,  7.44s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 82%|█████████████████████████████████▌       | 82/100 [10:46<02:14,  7.50s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 83%|██████████████████████████████████       | 83/100 [10:54<02:13,  7.84s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 84%|██████████████████████████████████▍      | 84/100 [11:03<02:07,  7.99s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 85%|██████████████████████████████████▊      | 85/100 [11:14<02:12,  8.83s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 86%|███████████████████████████████████▎     | 86/100 [11:21<01:57,  8.37s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 87%|███████████████████████████████████▋     | 87/100 [11:28<01:42,  7.92s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 88%|████████████████████████████████████     | 88/100 [11:37<01:38,  8.24s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 89%|████████████████████████████████████▍    | 89/100 [11:44<01:27,  7.95s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 90%|████████████████████████████████████▉    | 90/100 [11:52<01:19,  7.94s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 91%|█████████████████████████████████████▎   | 91/100 [11:58<01:05,  7.33s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 92%|█████████████████████████████████████▋   | 92/100 [12:05<00:58,  7.37s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 93%|██████████████████████████████████████▏  | 93/100 [12:15<00:57,  8.19s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 94%|██████████████████████████████████████▌  | 94/100 [12:26<00:53,  8.87s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 95%|██████████████████████████████████████▉  | 95/100 [12:35<00:44,  8.85s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 96%|███████████████████████████████████████▎ | 96/100 [12:42<00:33,  8.40s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 97%|███████████████████████████████████████▊ | 97/100 [12:51<00:26,  8.67s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 98%|████████████████████████████████████████▏| 98/100 [13:00<00:17,  8.73s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 99%|████████████████████████████████████████▌| 99/100 [13:10<00:08,  8.97s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "100%|████████████████████████████████████████| 100/100 [13:19<00:00,  8.94s/it]\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x21dc4cc0ba8>"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAECJJREFUeJzt3X+M5PVdx/HXS35YZNsDBMbrlXTbeDYia4/cSEia2NlCLXKx0ITGYkuOFLNVS23i/XNpG63WJlcjJf5BtBdBro12aSsEciDN9WSLJC11D68s10t7gGe943InAieLpLr69o/9opt1lu93Zr4z37k3z0cy2e+Pz8z3/WaZ1333O9/vdxwRAgCc+n6s6QIAAPUg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJI4fZQbO//882NycnKUm6zFSy+9pLPPPrvpMmpBL+MrUz/0Uq99+/Y9GxEXlI0baaBPTk5qfn5+lJusxdzcnDqdTtNl1IJexlemfuilXrb/qco4DrkAQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIjvVIUGFeT2+9vbNuHd2xpbNvIhT10AEiCQAeAJAh0AEiCQAeAJAh0AEiCQAeAJAh0AEiCQAeAJAh0AEiCQAeAJAh0AEiCQAeAJAh0AEiCQAeAJEoD3fbrbH/H9ndtH7D9+8Xyt9h+1PYh23fZPnP45QIA1lJlD/1Hkt4VEW+XtEnSVbYvl/Q5SbdGxEZJz0u6aXhlAgDKlAZ6LFssZs8oHiHpXZK+VizfJenaoVQIAKik0jF026fZ3i/phKQ9kp6S9EJELBVDjkjaMJwSAQBVOCKqD7bPkXSPpN+V9BcR8dPF8oskPRARU12eMyNpRpJardbm2dnZOuoeqcXFRU1MTDRdRi3opbuFoydreZ1+TG1YJ4nfzbgah16mp6f3RUS7bFxP3ykaES/YnpN0uaRzbJ9e7KW/SdIzazxnp6SdktRut6PT6fSyybEwNzenU7Hubuiluxub/E7RD3Yk8bsZV6dSL1XOcrmg2DOX7bMkXSnpoKSHJF1XDNsq6d5hFQkAKFdlD329pF22T9PyPwBfiYjdtr8nadb2H0r6B0m3D7FOAECJ0kCPiMclXdpl+dOSLhtGUQCA3nGlKAAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBIEOgAkQaADQBKlgW77ItsP2T5o+4DtjxfLP237qO39xePq4ZcLAFjL6RXGLEnaFhGP2X69pH229xTrbo2IPx5eeQCAqkoDPSKOSTpWTL9o+6CkDcMuDADQm56OoduelHSppEeLRTfbftz2HbbPrbk2AEAPHBHVBtoTkr4p6bMRcbftlqRnJYWkz0haHxEf7vK8GUkzktRqtTbPzs7WVfvILC4uamJioukyakEv3S0cPVnL6/RjasM6SfxuxtU49DI9Pb0vItpl4yoFuu0zJO2W9PWI+HyX9ZOSdkfEJa/2Ou12O+bn50u3N27m5ubU6XSaLqMW9NLd5Pb7a3mdfhzesUUSv5txNQ692K4U6FXOcrGk2yUdXBnmttevGPY+SU/0UygAoB5VznJ5h6QbJC3Y3l8s+4Sk621v0vIhl8OSPjKUCgEAlVQ5y+URSe6y6oH6ywEA9IsrRQEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIo/ZJoAMM1uf1+SdK2qSXdWEyPwuEdW0a2LYwGe+gAkASBDgBJlAa67YtsP2T7oO0Dtj9eLD/P9h7bh4qf5w6/XADAWqrsoS9J2hYRPyvpckkftX2xpO2S9kbERkl7i3kAQENKAz0ijkXEY8X0i5IOStog6RpJu4phuyRdO6wiAQDlHBHVB9uTkh6WdImkH0bEOSvWPR8R/++wi+0ZSTOS1Gq1Ns/Ozg5Y8ugtLi5qYmKi6TJqQS/dLRw9WcvrDKJ1lnT85dFtb2rDuqG9Nv+f1Wt6enpfRLTLxlUOdNsTkr4p6bMRcbftF6oE+krtdjvm5+crbW+czM3NqdPpNF1GLeilu8kRni64lm1TS7plYXRnEg/ztEX+P6uX7UqBXuksF9tnSPprSX8ZEXcXi4/bXl+sXy/pRL/FAgAGV+UsF0u6XdLBiPj8ilX3SdpaTG+VdG/95QEAqqry9907JN0gacH2/mLZJyTtkPQV2zdJ+qGk9w+nRABAFaWBHhGPSPIaq6+otxwAQL+4UhQAkiDQASAJAh0AkiDQASAJAh0AkiDQASAJAh0AkiDQASAJAh0AkiDQASAJAh0AkiDQASAJAh0AkiDQASAJAh0AkiDQASAJAh0AkiDQASAJAh0AkqjyJdEAEprcfv/QXnvb1JJufJXXP7xjy9C2/VrGHjoAJEGgA0ASpYFu+w7bJ2w/sWLZp20ftb2/eFw93DIBAGWq7KHfKemqLstvjYhNxeOBessCAPSqNNAj4mFJz42gFgDAAAY5hn6z7ceLQzLn1lYRAKAvjojyQfakpN0RcUkx35L0rKSQ9BlJ6yPiw2s8d0bSjCS1Wq3Ns7OztRQ+SouLi5qYmGi6jFrQS3cLR0/W8jqDaJ0lHX+56SrqUdbL1IZ1oytmQOPwnpment4XEe2ycX0FetV1q7Xb7Zifny/d3riZm5tTp9Npuoxa0Et3wzwnu6ptU0u6ZSHHpSFlvZxK56GPw3vGdqVA7+uQi+31K2bfJ+mJtcYCAEajdHfA9pcldSSdb/uIpN+T1LG9ScuHXA5L+sgQawQAVFAa6BFxfZfFtw+hFgDAALhSFACSINABIAkCHQCSINABIAkCHQCSINABIAkCHQCSyHGdMdLo5RL8sq85A15r2EMHgCQIdABIgkAHgCQIdABIgkAHgCQIdABIgkAHgCQIdABIgkAHgCQIdABIgkAHgCQIdABIgkAHgCQIdABIojTQbd9h+4TtJ1YsO8/2HtuHip/nDrdMAECZKnvod0q6atWy7ZL2RsRGSXuLeQBAg0oDPSIelvTcqsXXSNpVTO+SdG3NdQEAetTvMfRWRByTpOLnhfWVBADohyOifJA9KWl3RFxSzL8QEeesWP98RHQ9jm57RtKMJLVarc2zs7M1lD1ai4uLmpiYaLqMWox7LwtHT1Ye2zpLOv7yEIsZsUz9lPUytWHd6IoZ0Di8Z6anp/dFRLtsXL/fKXrc9vqIOGZ7vaQTaw2MiJ2SdkpSu92OTqfT5yabMzc3p1Ox7m7GvZdeviN029SSblnI87W4mfop6+XwBzujK2ZA4/6eWanfQy73SdpaTG+VdG895QAA+lXltMUvS/qWpLfZPmL7Jkk7JL3b9iFJ7y7mAQANKv37LiKuX2PVFTXXAgAYAFeKAkASOT6BQa0me/hgEsD4YA8dAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJIg0AEgCQIdAJI4fZAn2z4s6UVJ/yVpKSLadRQFAOjdQIFemI6IZ2t4HQDAADjkAgBJOCL6f7L9j5KelxSSvhARO7uMmZE0I0mtVmvz7Oxs39tryuLioiYmJpouoxZVelk4enJE1QymdZZ0/OWmq6hPpn7KepnasG50xQxoHN7/09PT+6oc0h400N8YEc/YvlDSHkkfi4iH1xrfbrdjfn6+7+01ZW5uTp1Op+kyalGll8nt94+mmAFtm1rSLQt1HDUcD5n6Kevl8I4tI6xmMOPw/rddKdAHOuQSEc8UP09IukfSZYO8HgCgf30Huu2zbb/+lWlJvyTpiboKAwD0ZpC/71qS7rH9yuv8VUQ8WEtVAICe9R3oEfG0pLfXWAsAYACctggASRDoAJAEgQ4ASRDoAJAEgQ4ASRDoAJAEgQ4ASeS4cQQAVNDPfYq2TS3pxhrubzSK+9ewhw4ASRDoAJAEgQ4ASRDoAJAEgQ4ASZwyZ7k0+S06d151dmPbBoCq2EMHgCQIdABIgkAHgCQIdABIgkAHgCQIdABI4pQ5bbFJC0dP1nJznnFQ142GgEE0eRpyZuyhA0ASAwW67atsf9/2k7a311UUAKB3fQe67dMk3SbplyVdLOl62xfXVRgAoDeD7KFfJunJiHg6Iv5D0qyka+opCwDQq0ECfYOkf14xf6RYBgBogCOivyfa75f0noj49WL+BkmXRcTHVo2bkTRTzL5N0vf7L7cx50t6tukiakIv4ytTP/RSrzdHxAVlgwY5bfGIpItWzL9J0jOrB0XETkk7B9hO42zPR0S76TrqQC/jK1M/9NKMQQ65/L2kjbbfYvtMSR+QdF89ZQEAetX3HnpELNm+WdLXJZ0m6Y6IOFBbZQCAngx0pWhEPCDpgZpqGWen9CGjVehlfGXqh14a0PeHogCA8cKl/wCQBIHehe3zbO+xfaj4eW6XMZtsf8v2AduP2/7VJmotU6WXYtyDtl+wvXvUNZYpu8WE7R+3fVex/lHbk6OvspoKvfyi7cdsL9m+rokae1Ghn9+x/b3iPbLX9pubqLOKCr38hu0F2/ttPzKWV8ZHBI9VD0l/JGl7Mb1d0ue6jPkZSRuL6TdKOibpnKZr76eXYt0Vkn5F0u6ma15V12mSnpL0VklnSvqupItXjfktSX9WTH9A0l1N1z1AL5OSfl7SFyVd13TNNfQzLekniunfPMV/N29YMf1eSQ82XffqB3vo3V0jaVcxvUvStasHRMQPIuJQMf2MpBOSSk/8b0BpL5IUEXslvTiqonpQ5RYTK3v8mqQrbHuENVZV2ktEHI6IxyX9dxMF9qhKPw9FxL8Xs9/W8vUq46hKL/+2YvZsSWP3ASSB3l0rIo5JUvHzwlcbbPsyLf+r/tQIautVT72MoSq3mPjfMRGxJOmkpJ8cSXW9yXa7jF77uUnS3wy1ov5V6sX2R20/peW/fH97RLVV9pr9ggvb35D0U11WfbLH11kv6UuStkZEI3tVdfUyprrtaa/eM6oyZhycKnVWVbkf2x+S1Jb0zqFW1L9KvUTEbZJus/1rkj4laeuwC+vFazbQI+LKtdbZPm57fUQcKwL7xBrj3iDpfkmfiohvD6nUUnX0Msaq3GLilTFHbJ8uaZ2k50ZTXk8q3S7jFFKpH9tXannn4p0R8aMR1darXn83s5L+dKgV9YFDLt3dp//7l3erpHtXDyhud3CPpC9GxFdHWFuvSnsZc1VuMbGyx+sk/W0Un1yNmWy3yyjtx/alkr4g6b0RMc47E1V62bhidoukQyOsr5qmP5Udx4eWj7/u1fIvbK+k84rlbUl/Xkx/SNJ/Stq/4rGp6dr76aWY/ztJ/yLpZS3vrbyn6dpX1Ha1pB9o+TOKTxbL/kDLISFJr5P0VUlPSvqOpLc2XfMAvfxC8d//JUn/KulA0zUP2M83JB1f8R65r+maB+jlTyQdKPp4SNLPNV3z6gdXigJAEhxyAYAkCHQASIJAB4AkCHQASIJAB4AkCHQASIJAB4AkCHQASOJ/ABCsBYvOUQL+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bootstraps = 100\n",
    "ates = pd.DataFrame([measure_treatment_effect(X_c, Z_c, y, MLPRegressor, {\"hidden_layer_sizes\": (100, )}) for i in tqdm(range(bootstraps))], columns=['ATE'])\n",
    "ates['ATE'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}