Medium_Files/hyp_tune.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "arSaacTbNR0_"
   },
   "source": [
    "<a id=000> </a>\n",
    "# <center style='color:blue;background:yellow'> Hyperparameters Optimization with Bayesian Optimization </center>\n",
    "Author:  [**Dayal Chand Aichara**](https://www.linkedin.com/in/dcaichara/) <div style=\"text-align:left\">Date: 16-08-2019 </div>\n",
    "Last updated on: 16-11-2020\n",
    "***  \n",
    "There are three steps to optimize hyperparameters with bayesian optimization.\n",
    "1. Define Objective Function  <br>\n",
    "    Write a function which has model and its output which has be maximized or minimized. \n",
    "2. Define Parameters Search Space (Range) <br>\n",
    "    Write a domain space of parameters in which parameters have to be optimized.\n",
    "3. Define Bayesian Optimization and Optimize.   <br>\n",
    "    Put model function and domain space in Bayesian Optimization function and optimize parameters.\n",
    "\n",
    "\n",
    "***\n",
    "### Install <span style='color:blue;background:orange;font-family:romon;font-size:25px'>[**bayesian-optimization**](https://github.com/fmfn/BayesianOptimization) </span> python package via pip.\n",
    "`pip install bayesian-optimization`\n",
    "\n",
    "##  Notebook Content\n",
    "1. [Import Libraries](#0)\n",
    "1. [Data](#1)\n",
    "1. [Simple Example](#2)\n",
    "1. [LGBM](#3)\n",
    "1. [CatBoost](#4)\n",
    "1. [XGBoost](#5)\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UwfBpvf4NR1B"
   },
   "source": [
    "<a id=0> </a>\n",
    "##  <span style='color:red;background:gray'>1. </span> <span style='color:blue;background:orange'>Import Libraries </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nwbV_v_ZNR1C"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import lightgbm as lgb\n",
    "import catboost as cgb\n",
    "import xgboost as xgb\n",
    "from bayes_opt import BayesianOptimization\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.metrics import r2_score\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rT8jv2JSNR1K"
   },
   "source": [
    "<a id=1> </a>\n",
    "##  <span style='color:red;background:gray'>2. </span> <span style='color:blue;background:orange'>Data </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BA850_WENR1L"
   },
   "outputs": [],
   "source": [
    "boston=load_boston()\n",
    "X =pd.DataFrame(boston.data,columns=boston.feature_names)\n",
    "y = boston.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "wR2cQCS2NR1O",
    "outputId": "3c69d3fb-9d5d-4f0b-b4c7-9f6dc22c38fc"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX  ...    TAX  PTRATIO       B  LSTAT  Price\n",
       "0  0.00632  18.0   2.31   0.0  0.538  ...  296.0     15.3  396.90   4.98   24.0\n",
       "1  0.02731   0.0   7.07   0.0  0.469  ...  242.0     17.8  396.90   9.14   21.6\n",
       "2  0.02729   0.0   7.07   0.0  0.469  ...  242.0     17.8  392.83   4.03   34.7\n",
       "3  0.03237   0.0   2.18   0.0  0.458  ...  222.0     18.7  394.63   2.94   33.4\n",
       "4  0.06905   0.0   2.18   0.0  0.458  ...  222.0     18.7  396.90   5.33   36.2\n",
       "\n",
       "[5 rows x 14 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df= X\n",
    "df['Price'] = y\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "x6eDpnHANR1V",
    "outputId": "c4fb0c7a-f816-4866-a2b1-816315fa99c8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 14)"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kAwQkPiTNR1Y"
   },
   "source": [
    "### Data has 14 columns in which 13 are features and <span style='background:blue;color:gray'>**last column is Price**</span>. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RE_vC6BkNR1c"
   },
   "source": [
    "<a id=2> </a>\n",
    "##  <span style='color:red;background:gray'>3. </span> <span style='color:blue;background:orange'>Simple Example </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-3NCu1QJNR1d"
   },
   "outputs": [],
   "source": [
    "# Define objective function\n",
    "def simple_fx(x, y, z ):\n",
    "    return -x ** 2 - (y - 1) -z** 2 + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "y0jlnrORNR1f"
   },
   "outputs": [],
   "source": [
    "# Search Space\n",
    "pds = {'x': (1, 4), 'y': (-3, 3), 'z': (1,6)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "id": "YGtyTWdUNR1j",
    "outputId": "a8a99fb1-3212-44af-d12d-6e7ba5e781f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   |     x     |     y     |     z     |\n",
      "-------------------------------------------------------------\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m-5.39    \u001b[0m | \u001b[0m 2.251   \u001b[0m | \u001b[0m 1.322   \u001b[0m | \u001b[0m 1.001   \u001b[0m |\n",
      "| \u001b[95m 2       \u001b[0m | \u001b[95m-1.654   \u001b[0m | \u001b[95m 1.907   \u001b[0m | \u001b[95m-2.119   \u001b[0m | \u001b[95m 1.462   \u001b[0m |\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m-8.406   \u001b[0m | \u001b[0m 1.559   \u001b[0m | \u001b[0m-0.9266  \u001b[0m | \u001b[0m 2.984   \u001b[0m |\n",
      "| \u001b[0m 4       \u001b[0m | \u001b[0m-11.46   \u001b[0m | \u001b[0m 2.593   \u001b[0m | \u001b[0m 0.8625  \u001b[0m | \u001b[0m 2.424   \u001b[0m |\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m-32.77   \u001b[0m | \u001b[0m 3.861   \u001b[0m | \u001b[0m 0.963   \u001b[0m | \u001b[0m 4.348   \u001b[0m |\n",
      "| \u001b[0m 6       \u001b[0m | \u001b[0m-2.077   \u001b[0m | \u001b[0m 1.857   \u001b[0m | \u001b[0m-1.913   \u001b[0m | \u001b[0m 1.594   \u001b[0m |\n",
      "| \u001b[95m 7       \u001b[0m | \u001b[95m-1.186   \u001b[0m | \u001b[95m 1.729   \u001b[0m | \u001b[95m-2.236   \u001b[0m | \u001b[95m 1.559   \u001b[0m |\n",
      "| \u001b[95m 8       \u001b[0m | \u001b[95m 0.02719 \u001b[0m | \u001b[95m 1.519   \u001b[0m | \u001b[95m-2.223   \u001b[0m | \u001b[95m 1.375   \u001b[0m |\n",
      "| \u001b[95m 9       \u001b[0m | \u001b[95m 1.4     \u001b[0m | \u001b[95m 1.252   \u001b[0m | \u001b[95m-2.199   \u001b[0m | \u001b[95m 1.11    \u001b[0m |\n",
      "| \u001b[95m 10      \u001b[0m | \u001b[95m 2.41    \u001b[0m | \u001b[95m 1.0     \u001b[0m | \u001b[95m-2.41    \u001b[0m | \u001b[95m 1.0     \u001b[0m |\n",
      "| \u001b[95m 11      \u001b[0m | \u001b[95m 2.746   \u001b[0m | \u001b[95m 1.0     \u001b[0m | \u001b[95m-2.746   \u001b[0m | \u001b[95m 1.0     \u001b[0m |\n",
      "| \u001b[95m 12      \u001b[0m | \u001b[95m 2.99    \u001b[0m | \u001b[95m 1.0     \u001b[0m | \u001b[95m-2.99    \u001b[0m | \u001b[95m 1.0     \u001b[0m |\n",
      "| \u001b[0m 13      \u001b[0m | \u001b[0m 2.622   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m-3.0     \u001b[0m | \u001b[0m 1.174   \u001b[0m |\n",
      "=============================================================\n"
     ]
    }
   ],
   "source": [
    "# optimization function and optimization\n",
    "optimizer = BayesianOptimization(f=simple_fx,\n",
    "                                 pbounds=pds,\n",
    "                                 random_state=1)\n",
    "optimizer.maximize(init_points=3,n_iter=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "kAOaOgSyNR1m",
    "outputId": "879487ed-56a5-4981-d06f-004b29b03356"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'params': {'x': 1.0, 'y': -2.9903855634307375, 'z': 1.0},\n",
       " 'target': 2.9903855634307375}"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check best results\n",
    "optimizer.max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 765
    },
    "colab_type": "code",
    "id": "X7slur3wNR1s",
    "outputId": "f47e2b77-de57-4ada-c4f1-e3140a4a789a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'params': {'x': 2.251066014107722,\n",
       "   'y': 1.3219469606529488,\n",
       "   'z': 1.0005718740867244},\n",
       "  'target': -5.390389235737196},\n",
       " {'params': {'x': 1.9069977178955193,\n",
       "   'y': -2.119464655097322,\n",
       "   'z': 1.461692973843989},\n",
       "  'target': -1.653721990746281},\n",
       " {'params': {'x': 1.5587806341330128,\n",
       "   'y': -0.9266356377417138,\n",
       "   'z': 2.9838373711533497},\n",
       "  'target': -8.406446885097736},\n",
       " {'params': {'x': 2.593123135025164,\n",
       "   'y': 0.8625228467524133,\n",
       "   'z': 2.423543047259528},\n",
       "  'target': -11.460371342075145},\n",
       " {'params': {'x': 3.8608951135676843,\n",
       "   'y': 0.9630016393643439,\n",
       "   'z': 4.347945130143197},\n",
       "  'target': -32.77413957207111},\n",
       " {'params': {'x': 1.8573822765338168,\n",
       "   'y': -1.9128784857090562,\n",
       "   'z': 1.5938529307268505},\n",
       "  'target': -2.0773576002594583},\n",
       " {'params': {'x': 1.7294837840933757,\n",
       "   'y': -2.236381944166488,\n",
       "   'z': 1.5593432461249863},\n",
       "  'target': -1.1862835745110636},\n",
       " {'params': {'x': 1.5186870970142565,\n",
       "   'y': -2.222855440130697,\n",
       "   'z': 1.3745031039706659},\n",
       "  'target': 0.027186158668111693},\n",
       " {'params': {'x': 1.2515310401872772,\n",
       "   'y': -2.199303100389387,\n",
       "   'z': 1.1104802247196097},\n",
       "  'target': 1.3998068263438241},\n",
       " {'params': {'x': 1.0, 'y': -2.409700117319973, 'z': 1.0},\n",
       "  'target': 2.409700117319973},\n",
       " {'params': {'x': 1.0, 'y': -2.7457437303883103, 'z': 1.0},\n",
       "  'target': 2.7457437303883103},\n",
       " {'params': {'x': 1.0, 'y': -2.9903855634307375, 'z': 1.0},\n",
       "  'target': 2.9903855634307375},\n",
       " {'params': {'x': 1.0, 'y': -3.0, 'z': 1.1740392185564508},\n",
       "  'target': 2.6216319132913584}]"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Get search history\n",
    "optimizer.res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LlqK4KA-NR1u"
   },
   "source": [
    "<a id=3> </a>\n",
    "##  <span style='color:red;background:gray'>4. </span> <span style='color:blue;background:orange'>LightGBM </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cStXPkPdNR1v"
   },
   "outputs": [],
   "source": [
    "dtrain = lgb.Dataset(data=X, label=y)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iTKdAj4dNR1y"
   },
   "outputs": [],
   "source": [
    "def lgb_r2_score(preds, dtrain):\n",
    "    labels = dtrain.get_label()\n",
    "    return 'r2', r2_score(labels, preds), True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7OokIZUcNR10"
   },
   "outputs": [],
   "source": [
    "# Objective Function\n",
    "def hyp_lgbm(num_leaves, feature_fraction, bagging_fraction, max_depth, min_split_gain, min_child_weight):\n",
    "      \n",
    "        params = {'application':'regression','num_iterations': 200,\n",
    "                  'learning_rate':0.05, 'early_stopping_round':50,\n",
    "                  'metric':'lgb_r2_score'} # Default parameters\n",
    "        params[\"num_leaves\"] = int(round(num_leaves))\n",
    "        params['feature_fraction'] = max(min(feature_fraction, 1), 0)\n",
    "        params['bagging_fraction'] = max(min(bagging_fraction, 1), 0)\n",
    "        params['max_depth'] = int(round(max_depth))\n",
    "        params['min_split_gain'] = min_split_gain\n",
    "        params['min_child_weight'] = min_child_weight\n",
    "        cv_results = lgb.cv(params, dtrain, nfold=5, seed=101,categorical_feature=[], stratified=False,\n",
    "                            verbose_eval =None, feval=lgb_r2_score)\n",
    "        # print(cv_results)\n",
    "        return np.max(cv_results['r2-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "j2f_MQnPNR13"
   },
   "outputs": [],
   "source": [
    "# Domain space-- Range of hyperparameters \n",
    "pds = {'num_leaves': (80, 100),\n",
    "          'feature_fraction': (0.1, 0.9),\n",
    "          'bagging_fraction': (0.8, 1),\n",
    "          'max_depth': (17, 25),\n",
    "          'min_split_gain': (0.001, 0.1),\n",
    "          'min_child_weight': (10, 25)\n",
    "          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "colab_type": "code",
    "id": "HUF2FKGgNR16",
    "outputId": "a5806555-1097-467a-f50d-f14df23bbf0e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   | baggin... | featur... | max_depth | min_ch... | min_sp... | num_le... |\n",
      "-------------------------------------------------------------------------------------------------\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 0.9766  \u001b[0m | \u001b[0m 0.9838  \u001b[0m | \u001b[0m 0.6138  \u001b[0m | \u001b[0m 23.03   \u001b[0m | \u001b[0m 12.09   \u001b[0m | \u001b[0m 0.009645\u001b[0m | \u001b[0m 95.76   \u001b[0m |\n",
      "| \u001b[0m 2       \u001b[0m | \u001b[0m 0.976   \u001b[0m | \u001b[0m 0.8652  \u001b[0m | \u001b[0m 0.5329  \u001b[0m | \u001b[0m 18.92   \u001b[0m | \u001b[0m 18.18   \u001b[0m | \u001b[0m 0.04065 \u001b[0m | \u001b[0m 94.3    \u001b[0m |\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m 0.976   \u001b[0m | \u001b[0m 0.9673  \u001b[0m | \u001b[0m 0.5708  \u001b[0m | \u001b[0m 19.37   \u001b[0m | \u001b[0m 14.22   \u001b[0m | \u001b[0m 0.07085 \u001b[0m | \u001b[0m 88.45   \u001b[0m |\n",
      "| \u001b[95m 4       \u001b[0m | \u001b[95m 0.9848  \u001b[0m | \u001b[95m 0.8115  \u001b[0m | \u001b[95m 0.6976  \u001b[0m | \u001b[95m 20.62   \u001b[0m | \u001b[95m 12.64   \u001b[0m | \u001b[95m 0.005888\u001b[0m | \u001b[95m 85.85   \u001b[0m |\n",
      "| \u001b[95m 5       \u001b[0m | \u001b[95m 0.9848  \u001b[0m | \u001b[95m 0.8134  \u001b[0m | \u001b[95m 0.7009  \u001b[0m | \u001b[95m 17.51   \u001b[0m | \u001b[95m 16.48   \u001b[0m | \u001b[95m 0.03705 \u001b[0m | \u001b[95m 83.04   \u001b[0m |\n",
      "| \u001b[0m 6       \u001b[0m | \u001b[0m 0.9725  \u001b[0m | \u001b[0m 0.9646  \u001b[0m | \u001b[0m 0.5965  \u001b[0m | \u001b[0m 24.87   \u001b[0m | \u001b[0m 24.75   \u001b[0m | \u001b[0m 0.0666  \u001b[0m | \u001b[0m 80.22   \u001b[0m |\n",
      "| \u001b[95m 7       \u001b[0m | \u001b[95m 0.9855  \u001b[0m | \u001b[95m 0.9372  \u001b[0m | \u001b[95m 0.7901  \u001b[0m | \u001b[95m 24.98   \u001b[0m | \u001b[95m 24.48   \u001b[0m | \u001b[95m 0.08313 \u001b[0m | \u001b[95m 99.67   \u001b[0m |\n",
      "| \u001b[0m 8       \u001b[0m | \u001b[0m 0.8837  \u001b[0m | \u001b[0m 0.8     \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 17.0    \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 0.001   \u001b[0m | \u001b[0m 80.0    \u001b[0m |\n",
      "| \u001b[95m 9       \u001b[0m | \u001b[95m 0.9868  \u001b[0m | \u001b[95m 0.8     \u001b[0m | \u001b[95m 0.9     \u001b[0m | \u001b[95m 17.0    \u001b[0m | \u001b[95m 25.0    \u001b[0m | \u001b[95m 0.001   \u001b[0m | \u001b[95m 86.97   \u001b[0m |\n",
      "| \u001b[95m 10      \u001b[0m | \u001b[95m 0.9921  \u001b[0m | \u001b[95m 0.8     \u001b[0m | \u001b[95m 0.9     \u001b[0m | \u001b[95m 25.0    \u001b[0m | \u001b[95m 18.44   \u001b[0m | \u001b[95m 0.004544\u001b[0m | \u001b[95m 89.58   \u001b[0m |\n",
      "| \u001b[0m 11      \u001b[0m | \u001b[0m 0.9867  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 0.9     \u001b[0m | \u001b[0m 17.0    \u001b[0m | \u001b[0m 25.0    \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 100.0   \u001b[0m |\n",
      "| \u001b[0m 12      \u001b[0m | \u001b[0m 0.9872  \u001b[0m | \u001b[0m 0.8725  \u001b[0m | \u001b[0m 0.7855  \u001b[0m | \u001b[0m 24.99   \u001b[0m | \u001b[0m 13.98   \u001b[0m | \u001b[0m 0.08693 \u001b[0m | \u001b[0m 80.22   \u001b[0m |\n",
      "| \u001b[0m 13      \u001b[0m | \u001b[0m 0.992   \u001b[0m | \u001b[0m 0.8     \u001b[0m | \u001b[0m 0.9     \u001b[0m | \u001b[0m 17.04   \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 0.04957 \u001b[0m | \u001b[0m 100.0   \u001b[0m |\n",
      "| \u001b[0m 14      \u001b[0m | \u001b[0m 0.9906  \u001b[0m | \u001b[0m 0.864   \u001b[0m | \u001b[0m 0.8423  \u001b[0m | \u001b[0m 24.86   \u001b[0m | \u001b[0m 10.91   \u001b[0m | \u001b[0m 0.03088 \u001b[0m | \u001b[0m 99.95   \u001b[0m |\n",
      "| \u001b[0m 15      \u001b[0m | \u001b[0m 0.9867  \u001b[0m | \u001b[0m 0.9287  \u001b[0m | \u001b[0m 0.8935  \u001b[0m | \u001b[0m 17.46   \u001b[0m | \u001b[0m 24.21   \u001b[0m | \u001b[0m 0.09273 \u001b[0m | \u001b[0m 80.23   \u001b[0m |\n",
      "| \u001b[0m 16      \u001b[0m | \u001b[0m 0.9921  \u001b[0m | \u001b[0m 0.8896  \u001b[0m | \u001b[0m 0.8989  \u001b[0m | \u001b[0m 21.46   \u001b[0m | \u001b[0m 19.71   \u001b[0m | \u001b[0m 0.009563\u001b[0m | \u001b[0m 81.69   \u001b[0m |\n",
      "| \u001b[0m 17      \u001b[0m | \u001b[0m 0.9921  \u001b[0m | \u001b[0m 0.9052  \u001b[0m | \u001b[0m 0.8794  \u001b[0m | \u001b[0m 17.02   \u001b[0m | \u001b[0m 16.44   \u001b[0m | \u001b[0m 0.00265 \u001b[0m | \u001b[0m 99.8    \u001b[0m |\n",
      "| \u001b[0m 18      \u001b[0m | \u001b[0m 0.9905  \u001b[0m | \u001b[0m 0.8298  \u001b[0m | \u001b[0m 0.847   \u001b[0m | \u001b[0m 24.85   \u001b[0m | \u001b[0m 10.25   \u001b[0m | \u001b[0m 0.07434 \u001b[0m | \u001b[0m 88.37   \u001b[0m |\n",
      "| \u001b[0m 19      \u001b[0m | \u001b[0m 0.9315  \u001b[0m | \u001b[0m 0.811   \u001b[0m | \u001b[0m 0.1449  \u001b[0m | \u001b[0m 24.89   \u001b[0m | \u001b[0m 15.59   \u001b[0m | \u001b[0m 0.02002 \u001b[0m | \u001b[0m 84.6    \u001b[0m |\n",
      "| \u001b[0m 20      \u001b[0m | \u001b[0m 0.992   \u001b[0m | \u001b[0m 0.8     \u001b[0m | \u001b[0m 0.9     \u001b[0m | \u001b[0m 17.0    \u001b[0m | \u001b[0m 10.53   \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 93.85   \u001b[0m |\n",
      "=================================================================================================\n"
     ]
    }
   ],
   "source": [
    "# Surrogate model\n",
    "optimizer = BayesianOptimization(hyp_lgbm, pds, random_state=77)\n",
    "                                  \n",
    "# Optimize\n",
    "optimizer.maximize(init_points=5, n_iter=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "id": "EL5dZ9ZgNR19",
    "outputId": "ff221a4d-6513-4e8b-836f-7d6647bfb720"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'params': {'bagging_fraction': 0.8,\n",
       "  'feature_fraction': 0.9,\n",
       "  'max_depth': 25.0,\n",
       "  'min_child_weight': 18.437809423883323,\n",
       "  'min_split_gain': 0.004544331379525425,\n",
       "  'num_leaves': 89.5835303879813},\n",
       " 'target': 0.9921383303951028}"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimizer.max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "colab_type": "code",
    "id": "oUBWJDKJfpyH",
    "outputId": "53c49993-dfe2-4c93-ffe0-70e3ad89f78e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   | baggin... | featur... | max_depth | min_ch... | min_sp... | num_le... |\n",
      "-------------------------------------------------------------------------------------------------\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 0.9853  \u001b[0m | \u001b[0m 0.9304  \u001b[0m | \u001b[0m 0.6844  \u001b[0m | \u001b[0m 14.93   \u001b[0m | \u001b[0m 17.8    \u001b[0m | \u001b[0m 0.06796 \u001b[0m | \u001b[0m 66.09   \u001b[0m |\n",
      "| \u001b[95m 2       \u001b[0m | \u001b[95m 0.9917  \u001b[0m | \u001b[95m 0.9067  \u001b[0m | \u001b[95m 0.8264  \u001b[0m | \u001b[95m 17.43   \u001b[0m | \u001b[95m 13.84   \u001b[0m | \u001b[95m 0.02859 \u001b[0m | \u001b[95m 50.87   \u001b[0m |\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m 0.9862  \u001b[0m | \u001b[0m 0.8938  \u001b[0m | \u001b[0m 0.8533  \u001b[0m | \u001b[0m 19.0    \u001b[0m | \u001b[0m 23.03   \u001b[0m | \u001b[0m 0.03538 \u001b[0m | \u001b[0m 60.4    \u001b[0m |\n",
      "| \u001b[0m 4       \u001b[0m | \u001b[0m 0.9769  \u001b[0m | \u001b[0m 0.8164  \u001b[0m | \u001b[0m 0.5826  \u001b[0m | \u001b[0m 13.68   \u001b[0m | \u001b[0m 16.29   \u001b[0m | \u001b[0m 0.05168 \u001b[0m | \u001b[0m 52.17   \u001b[0m |\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m 0.9742  \u001b[0m | \u001b[0m 0.9952  \u001b[0m | \u001b[0m 0.5607  \u001b[0m | \u001b[0m 19.89   \u001b[0m | \u001b[0m 23.38   \u001b[0m | \u001b[0m 0.09741 \u001b[0m | \u001b[0m 58.82   \u001b[0m |\n",
      "| \u001b[95m 6       \u001b[0m | \u001b[95m 0.9933  \u001b[0m | \u001b[95m 0.8     \u001b[0m | \u001b[95m 0.9     \u001b[0m | \u001b[95m 23.0    \u001b[0m | \u001b[95m 10.0    \u001b[0m | \u001b[95m 0.001   \u001b[0m | \u001b[95m 70.0    \u001b[0m |\n",
      "| \u001b[0m 7       \u001b[0m | \u001b[0m 0.9851  \u001b[0m | \u001b[0m 0.9379  \u001b[0m | \u001b[0m 0.8013  \u001b[0m | \u001b[0m 22.84   \u001b[0m | \u001b[0m 24.89   \u001b[0m | \u001b[0m 0.02053 \u001b[0m | \u001b[0m 69.96   \u001b[0m |\n",
      "| \u001b[95m 8       \u001b[0m | \u001b[95m 0.9933  \u001b[0m | \u001b[95m 0.8137  \u001b[0m | \u001b[95m 0.8657  \u001b[0m | \u001b[95m 22.47   \u001b[0m | \u001b[95m 10.04   \u001b[0m | \u001b[95m 0.0861  \u001b[0m | \u001b[95m 57.23   \u001b[0m |\n",
      "| \u001b[0m 9       \u001b[0m | \u001b[0m 0.9866  \u001b[0m | \u001b[0m 0.8594  \u001b[0m | \u001b[0m 0.8999  \u001b[0m | \u001b[0m 13.17   \u001b[0m | \u001b[0m 24.69   \u001b[0m | \u001b[0m 0.09088 \u001b[0m | \u001b[0m 69.83   \u001b[0m |\n",
      "| \u001b[0m 10      \u001b[0m | \u001b[0m 0.9918  \u001b[0m | \u001b[0m 0.811   \u001b[0m | \u001b[0m 0.8371  \u001b[0m | \u001b[0m 13.95   \u001b[0m | \u001b[0m 10.26   \u001b[0m | \u001b[0m 0.04601 \u001b[0m | \u001b[0m 69.98   \u001b[0m |\n",
      "| \u001b[0m 11      \u001b[0m | \u001b[0m 0.9933  \u001b[0m | \u001b[0m 0.8888  \u001b[0m | \u001b[0m 0.8822  \u001b[0m | \u001b[0m 22.86   \u001b[0m | \u001b[0m 10.24   \u001b[0m | \u001b[0m 0.06043 \u001b[0m | \u001b[0m 50.15   \u001b[0m |\n",
      "| \u001b[95m 12      \u001b[0m | \u001b[95m 0.9933  \u001b[0m | \u001b[95m 0.8409  \u001b[0m | \u001b[95m 0.8975  \u001b[0m | \u001b[95m 13.0    \u001b[0m | \u001b[95m 10.13   \u001b[0m | \u001b[95m 0.07254 \u001b[0m | \u001b[95m 50.45   \u001b[0m |\n",
      "| \u001b[0m 13      \u001b[0m | \u001b[0m 0.9932  \u001b[0m | \u001b[0m 0.8472  \u001b[0m | \u001b[0m 0.8925  \u001b[0m | \u001b[0m 16.9    \u001b[0m | \u001b[0m 10.02   \u001b[0m | \u001b[0m 0.001145\u001b[0m | \u001b[0m 54.7    \u001b[0m |\n",
      "| \u001b[0m 14      \u001b[0m | \u001b[0m 0.9932  \u001b[0m | \u001b[0m 0.8229  \u001b[0m | \u001b[0m 0.8871  \u001b[0m | \u001b[0m 20.02   \u001b[0m | \u001b[0m 17.18   \u001b[0m | \u001b[0m 0.0609  \u001b[0m | \u001b[0m 69.63   \u001b[0m |\n",
      "| \u001b[0m 15      \u001b[0m | \u001b[0m 0.9932  \u001b[0m | \u001b[0m 0.9781  \u001b[0m | \u001b[0m 0.8987  \u001b[0m | \u001b[0m 22.98   \u001b[0m | \u001b[0m 14.25   \u001b[0m | \u001b[0m 0.02232 \u001b[0m | \u001b[0m 60.59   \u001b[0m |\n",
      "| \u001b[0m 16      \u001b[0m | \u001b[0m 0.9932  \u001b[0m | \u001b[0m 0.9871  \u001b[0m | \u001b[0m 0.8998  \u001b[0m | \u001b[0m 18.27   \u001b[0m | \u001b[0m 10.6    \u001b[0m | \u001b[0m 0.002164\u001b[0m | \u001b[0m 69.91   \u001b[0m |\n",
      "| \u001b[0m 17      \u001b[0m | \u001b[0m 0.9933  \u001b[0m | \u001b[0m 0.8097  \u001b[0m | \u001b[0m 0.8988  \u001b[0m | \u001b[0m 18.88   \u001b[0m | \u001b[0m 10.37   \u001b[0m | \u001b[0m 0.04609 \u001b[0m | \u001b[0m 50.01   \u001b[0m |\n",
      "| \u001b[0m 18      \u001b[0m | \u001b[0m 0.9932  \u001b[0m | \u001b[0m 0.8213  \u001b[0m | \u001b[0m 0.8963  \u001b[0m | \u001b[0m 15.16   \u001b[0m | \u001b[0m 10.13   \u001b[0m | \u001b[0m 0.09663 \u001b[0m | \u001b[0m 64.04   \u001b[0m |\n",
      "| \u001b[0m 19      \u001b[0m | \u001b[0m 0.9932  \u001b[0m | \u001b[0m 0.8709  \u001b[0m | \u001b[0m 0.8948  \u001b[0m | \u001b[0m 22.99   \u001b[0m | \u001b[0m 16.45   \u001b[0m | \u001b[0m 0.02483 \u001b[0m | \u001b[0m 50.01   \u001b[0m |\n",
      "| \u001b[0m 20      \u001b[0m | \u001b[0m 0.9932  \u001b[0m | \u001b[0m 0.8099  \u001b[0m | \u001b[0m 0.896   \u001b[0m | \u001b[0m 22.72   \u001b[0m | \u001b[0m 14.65   \u001b[0m | \u001b[0m 0.005227\u001b[0m | \u001b[0m 57.27   \u001b[0m |\n",
      "=================================================================================================\n"
     ]
    }
   ],
   "source": [
    "def bayesion_opt_lgbm(X=X, y=y, init_iter=3, n_iters=7, random_state=11, seed = 101, num_iterations = 200):\n",
    "  dtrain = lgb.Dataset(data=X, label=y)\n",
    "  def lgb_r2_score(preds, dtrain):\n",
    "      labels = dtrain.get_label()\n",
    "      return 'r2', r2_score(labels, preds), True\n",
    "  # Objective Function\n",
    "  def hyp_lgbm(num_leaves, feature_fraction, bagging_fraction, max_depth, min_split_gain, min_child_weight):\n",
    "        \n",
    "          params = {'application':'regression','num_iterations': num_iterations,\n",
    "                    'learning_rate':0.05, 'early_stopping_round':50,\n",
    "                    'metric':'lgb_r2_score'} # Default parameters\n",
    "          params[\"num_leaves\"] = int(round(num_leaves))\n",
    "          params['feature_fraction'] = max(min(feature_fraction, 1), 0)\n",
    "          params['bagging_fraction'] = max(min(bagging_fraction, 1), 0)\n",
    "          params['max_depth'] = int(round(max_depth))\n",
    "          params['min_split_gain'] = min_split_gain\n",
    "          params['min_child_weight'] = min_child_weight\n",
    "          cv_results = lgb.cv(params, dtrain, nfold=5, seed=seed,categorical_feature=[], stratified=False,\n",
    "                              verbose_eval =None, feval=lgb_r2_score)\n",
    "          # print(cv_results)\n",
    "          return np.max(cv_results['r2-mean'])\n",
    "  # Domain space-- Range of hyperparameters \n",
    "  pds = {'num_leaves': (50, 70),\n",
    "            'feature_fraction': (0.1, 0.9),\n",
    "            'bagging_fraction': (0.8, 1),\n",
    "            'max_depth': (13, 23),\n",
    "            'min_split_gain': (0.001, 0.1),\n",
    "            'min_child_weight': (10, 25)\n",
    "            }\n",
    "\n",
    "  # Surrogate model\n",
    "  optimizer = BayesianOptimization(hyp_lgbm, pds, random_state=random_state)\n",
    "                                    \n",
    "  # Optimize\n",
    "  optimizer.maximize(init_points=init_iter, n_iter=n_iters)\n",
    "\n",
    "bayesion_opt_lgbm(X=X, y=y, init_iter=5, n_iters=15, random_state=717, seed = 1011, num_iterations = 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "W1v2Yv8jNR2A"
   },
   "source": [
    "<a id=4> </a>\n",
    "##  <span style='color:red;background:gray'>5. </span> <span style='color:blue;background:orange'>CatBoost </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Oppm466UNR2B"
   },
   "outputs": [],
   "source": [
    "cat_features = []\n",
    "\n",
    "cv_dataset = cgb.Pool(data=X,\n",
    "                  label=y,\n",
    "                  cat_features=cat_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KTRkUvq1NR2G"
   },
   "outputs": [],
   "source": [
    "def hyp_cat(depth, bagging_temperature):\n",
    "    params = {\"iterations\": 300,\n",
    "              \"learning_rate\": 0.05,\n",
    "              \"eval_metric\": \"R2\",\n",
    "              \"loss_function\": \"RMSE\",\n",
    "              \"verbose\": False}\n",
    "    params[ \"depth\"] = int(round(depth))\n",
    "    params[\"bagging_temperature\"] = bagging_temperature\n",
    "\n",
    "    scores = cgb.cv(cv_dataset,\n",
    "                params,\n",
    "                fold_count=3)\n",
    "    return np.max(scores['test-R2-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "edyVKdw4NR2J"
   },
   "outputs": [],
   "source": [
    "pds = {'depth': (6, 10),\n",
    "          'bagging_temperature': (1,5),\n",
    "          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "colab_type": "code",
    "id": "2op74G9RNR2M",
    "outputId": "6e90af7e-d316-4f1d-eb0c-f54f74dfd562"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   | baggin... |   depth   |\n",
      "-------------------------------------------------\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 0.9481  \u001b[0m | \u001b[0m 3.174   \u001b[0m | \u001b[0m 7.113   \u001b[0m |\n",
      "| \u001b[0m 2       \u001b[0m | \u001b[0m 0.9275  \u001b[0m | \u001b[0m 2.698   \u001b[0m | \u001b[0m 9.379   \u001b[0m |\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m 0.936   \u001b[0m | \u001b[0m 1.019   \u001b[0m | \u001b[0m 6.486   \u001b[0m |\n",
      "| \u001b[95m 4       \u001b[0m | \u001b[95m 0.9592  \u001b[0m | \u001b[95m 5.0     \u001b[0m | \u001b[95m 6.0     \u001b[0m |\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m 0.9571  \u001b[0m | \u001b[0m 5.0     \u001b[0m | \u001b[0m 7.292   \u001b[0m |\n",
      "| \u001b[0m 6       \u001b[0m | \u001b[0m 0.9592  \u001b[0m | \u001b[0m 5.0     \u001b[0m | \u001b[0m 6.0     \u001b[0m |\n",
      "| \u001b[95m 7       \u001b[0m | \u001b[95m 0.9598  \u001b[0m | \u001b[95m 4.991   \u001b[0m | \u001b[95m 6.017   \u001b[0m |\n",
      "| \u001b[0m 8       \u001b[0m | \u001b[0m 0.9598  \u001b[0m | \u001b[0m 4.992   \u001b[0m | \u001b[0m 6.009   \u001b[0m |\n",
      "| \u001b[0m 9       \u001b[0m | \u001b[0m 0.9596  \u001b[0m | \u001b[0m 4.996   \u001b[0m | \u001b[0m 6.004   \u001b[0m |\n",
      "| \u001b[0m 10      \u001b[0m | \u001b[0m 0.9589  \u001b[0m | \u001b[0m 4.998   \u001b[0m | \u001b[0m 6.038   \u001b[0m |\n",
      "=================================================\n"
     ]
    }
   ],
   "source": [
    "# Surrogate model\n",
    "optimizer = BayesianOptimization(hyp_cat, pds, random_state=100)\n",
    "                                  \n",
    "# Optimize\n",
    "optimizer.maximize(init_points=3, n_iter=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "YLXcDMknNR2O",
    "outputId": "d76dd88f-b07d-4f33-b852-b020468b09c9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'params': {'bagging_temperature': 4.991446336204385,\n",
       "  'depth': 6.0172335892654525},\n",
       " 'target': 0.9598253840405738}"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimizer.max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6NpamoxzNR2S"
   },
   "source": [
    "<a id=5> </a>\n",
    "##  <span style='color:red;background:gray'>6. </span> <span style='color:blue;background:orange'>XGBoost </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ot-o6hwANR2S"
   },
   "outputs": [],
   "source": [
    "dtrain = xgb.DMatrix(X, y, feature_names=X.columns.values)\n",
    "def xgb_r2(preds, dtrain):\n",
    "    labels = dtrain.get_label()\n",
    "    return 'r2', r2_score(preds, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EGzGaT3rNR2V"
   },
   "outputs": [],
   "source": [
    "def hyp_xgb(max_depth, subsample, colsample_bytree,min_child_weight, gamma ):\n",
    "    params = {\n",
    "    'n_estimators': 300,\n",
    "    'eta': 0.05,\n",
    "    'objective': 'reg:linear',\n",
    "    'eval_metric':'mae',\n",
    "    'silent': 1\n",
    "     }\n",
    "    params['max_depth'] = int(round(max_depth))\n",
    "    params['subsample'] = max(min(subsample, 1), 0)\n",
    "    params['colsample_bytree'] = max(min(colsample_bytree, 1), 0)\n",
    "    params['min_child_weight'] = int(min_child_weight)\n",
    "    params['gamma'] = max(gamma, 0)\n",
    "    scores = xgb.cv(params, dtrain, num_boost_round=1000,verbose_eval=False, early_stopping_rounds=5, feval=xgb_r2, maximize=True, nfold=5)\n",
    "    return  scores['test-r2-mean'].iloc[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kDGnAqzGNR2X"
   },
   "outputs": [],
   "source": [
    "pds ={\n",
    "  'min_child_weight':(14, 20),\n",
    "  'gamma':(0, 5),\n",
    "  'subsample':(0.5, 1),\n",
    "  'colsample_bytree':(0.1, 1),\n",
    "  'max_depth': (6, 10)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "colab_type": "code",
    "id": "dW_Z8TSmNR2a",
    "outputId": "4bd1cd5f-db2e-408f-d52c-77c883faf369"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   | colsam... |   gamma   | max_depth | min_ch... | subsample |\n",
      "-------------------------------------------------------------------------------------\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 0.9732  \u001b[0m | \u001b[0m 0.4889  \u001b[0m | \u001b[0m 0.8711  \u001b[0m | \u001b[0m 6.684   \u001b[0m | \u001b[0m 18.97   \u001b[0m | \u001b[0m 0.7936  \u001b[0m |\n",
      "| \u001b[95m 2       \u001b[0m | \u001b[95m 0.9774  \u001b[0m | \u001b[95m 0.5134  \u001b[0m | \u001b[95m 4.113   \u001b[0m | \u001b[95m 9.286   \u001b[0m | \u001b[95m 15.84   \u001b[0m | \u001b[95m 0.6004  \u001b[0m |\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m 0.97    \u001b[0m | \u001b[0m 0.4629  \u001b[0m | \u001b[0m 4.739   \u001b[0m | \u001b[0m 8.71    \u001b[0m | \u001b[0m 17.65   \u001b[0m | \u001b[0m 0.8362  \u001b[0m |\n",
      "| \u001b[0m 4       \u001b[0m | \u001b[0m 0.8031  \u001b[0m | \u001b[0m 0.1072  \u001b[0m | \u001b[0m 1.683   \u001b[0m | \u001b[0m 7.429   \u001b[0m | \u001b[0m 16.92   \u001b[0m | \u001b[0m 0.9391  \u001b[0m |\n",
      "| \u001b[95m 5       \u001b[0m | \u001b[95m 0.9909  \u001b[0m | \u001b[95m 0.7794  \u001b[0m | \u001b[95m 3.152   \u001b[0m | \u001b[95m 7.506   \u001b[0m | \u001b[95m 17.54   \u001b[0m | \u001b[95m 0.8687  \u001b[0m |\n",
      "| \u001b[95m 6       \u001b[0m | \u001b[95m 0.9919  \u001b[0m | \u001b[95m 1.0     \u001b[0m | \u001b[95m 5.0     \u001b[0m | \u001b[95m 6.0     \u001b[0m | \u001b[95m 14.0    \u001b[0m | \u001b[95m 0.5     \u001b[0m |\n",
      "| \u001b[0m 7       \u001b[0m | \u001b[0m 0.9654  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 4.441e-1\u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 20.0    \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[0m 8       \u001b[0m | \u001b[0m 0.9654  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 5.0     \u001b[0m | \u001b[0m 6.0     \u001b[0m | \u001b[0m 20.0    \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[95m 9       \u001b[0m | \u001b[95m 0.9987  \u001b[0m | \u001b[95m 1.0     \u001b[0m | \u001b[95m 5.0     \u001b[0m | \u001b[95m 10.0    \u001b[0m | \u001b[95m 14.0    \u001b[0m | \u001b[95m 1.0     \u001b[0m |\n",
      "| \u001b[0m 10      \u001b[0m | \u001b[0m 0.9654  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 5.0     \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 20.0    \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[0m 11      \u001b[0m | \u001b[0m 0.9676  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 6.0     \u001b[0m | \u001b[0m 20.0    \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[0m 12      \u001b[0m | \u001b[0m 0.7965  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 2.482   \u001b[0m | \u001b[0m 7.936   \u001b[0m | \u001b[0m 20.0    \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[0m 13      \u001b[0m | \u001b[0m 0.9923  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 14.0    \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[0m 14      \u001b[0m | \u001b[0m 0.9923  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 6.0     \u001b[0m | \u001b[0m 14.0    \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[0m 15      \u001b[0m | \u001b[0m 0.9844  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 5.0     \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 16.82   \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[0m 16      \u001b[0m | \u001b[0m 0.9984  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 5.0     \u001b[0m | \u001b[0m 6.0     \u001b[0m | \u001b[0m 16.91   \u001b[0m | \u001b[0m 1.0     \u001b[0m |\n",
      "| \u001b[0m 17      \u001b[0m | \u001b[0m 0.998   \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 17.28   \u001b[0m | \u001b[0m 1.0     \u001b[0m |\n",
      "| \u001b[0m 18      \u001b[0m | \u001b[0m 0.9789  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 6.0     \u001b[0m | \u001b[0m 17.38   \u001b[0m | \u001b[0m 0.5     \u001b[0m |\n",
      "| \u001b[95m 19      \u001b[0m | \u001b[95m 0.9989  \u001b[0m | \u001b[95m 1.0     \u001b[0m | \u001b[95m 2.831   \u001b[0m | \u001b[95m 8.202   \u001b[0m | \u001b[95m 14.0    \u001b[0m | \u001b[95m 1.0     \u001b[0m |\n",
      "| \u001b[0m 20      \u001b[0m | \u001b[0m 0.9959  \u001b[0m | \u001b[0m 1.0     \u001b[0m | \u001b[0m 2.25    \u001b[0m | \u001b[0m 6.0     \u001b[0m | \u001b[0m 19.64   \u001b[0m | \u001b[0m 1.0     \u001b[0m |\n",
      "=====================================================================================\n"
     ]
    }
   ],
   "source": [
    "# Surrogate model\n",
    "optimizer = BayesianOptimization(hyp_xgb, pds, random_state=103)\n",
    "                                  \n",
    "# Optimize\n",
    "optimizer.maximize(init_points=5, n_iter=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "JhG3DcmNfq0Z",
    "outputId": "b71ea0a6-cea8-4934-d029-5b5428362cb2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'params': {'colsample_bytree': 1.0,\n",
       "  'gamma': 2.830706438799185,\n",
       "  'max_depth': 8.201814553535812,\n",
       "  'min_child_weight': 14.0,\n",
       "  'subsample': 1.0},\n",
       " 'target': 0.9989032}"
      ]
     },
     "execution_count": 29,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimizer.max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PirO-H4qNR2c"
   },
   "source": [
    "***\n",
    "## References: \n",
    "1. https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f <br>\n",
    "\n",
    "2. https://towardsdatascience.com/an-introductory-example-of-bayesian-optimization-in-python-with-hyperopt-aae40fff4ffo <br>\n",
    "\n",
    "3. https://medium.com/spikelab/hyperparameter-optimization-using-bayesian-optimization-f1f393dcd36d  <br>\n",
    "\n",
    "4. https://www.kaggle.com/omarito/xgboost-bayesianoptimization <br>\n",
    "\n",
    "5. https://github.com/fmfn/BayesianOptimization\n",
    "\n",
    "***\n",
    "\n",
    "\n",
    "\n",
    "<center><span style='color:red;background:pink;font-size:40px'>End of the Notebook </span> </center>\n",
    "\n",
    "***\n",
    "\n",
    "### <center style='color:blue;background:yellow'>  [GOT TO TOP](#000) </center>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "hyp_tune.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}