experiments.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pipeline as p\n",
    "import globals\n",
    "import pandas as pd\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accompanying paper: please click [[here]](https://doi.org/10.5194/amt-14-37-2021).\n",
    "### Please see the last section of this notebook (click [[here]](#Final-Results)) for final results and reproducibility checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 1: calibration results on the original datasets\n",
    "Please note that only 2 of the 12 datasets used in the accompanying paper, namely DD1(Jun) and DD1(Oct), are provided as a part of this repository in the directory [data/](data/). Only those results that rely only on these two datasets can be reproduced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialising global variables. For details, please refer to globals.py\n",
    "g1 = globals.My_Globals()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load a dataset and perform preprocessing\n",
    "**Note**: The calibration algorithms rely on train-test splits to obtain an unbiased estimate of the performance of various calibration algorithms. If you wish to reproduce the results in the paper, please use the train-test splits supplied for the 2 datasets DD1(Jun) and DD1(Oct) in the directory [perm/](perm/).\n",
    "\n",
    "To do so, please use ```new_permutations = False``` below. Using ```new_permutations = True``` instead, allows fresh train-test splits to be created which will then overwrite the supplied train-test splits. However, minor differences are to be expected in the results if fresh splits are used to conduct the experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets, datasets_norm = p.load_data( g1, new_permutations = False )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ref. O3(ppb)</th>\n",
       "      <th>Ref. NO2(ppb)</th>\n",
       "      <th>Temp(C)</th>\n",
       "      <th>RH(%)</th>\n",
       "      <th>no2op1(mV)</th>\n",
       "      <th>no2op2(mV)</th>\n",
       "      <th>o3op1(mV)</th>\n",
       "      <th>o3op2(mV)</th>\n",
       "      <th>no2op1 - no2op2(mV)</th>\n",
       "      <th>o3op1 - o3op2(mV)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2340</th>\n",
       "      <td>22.223</td>\n",
       "      <td>16.407</td>\n",
       "      <td>0.239336</td>\n",
       "      <td>-1.982867</td>\n",
       "      <td>1.415799</td>\n",
       "      <td>1.372104</td>\n",
       "      <td>1.322321</td>\n",
       "      <td>1.403552</td>\n",
       "      <td>0.809485</td>\n",
       "      <td>0.150671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19549</th>\n",
       "      <td>19.247</td>\n",
       "      <td>11.387</td>\n",
       "      <td>-0.008013</td>\n",
       "      <td>0.536193</td>\n",
       "      <td>0.274850</td>\n",
       "      <td>0.302993</td>\n",
       "      <td>0.143411</td>\n",
       "      <td>0.306001</td>\n",
       "      <td>-0.133636</td>\n",
       "      <td>-0.852900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57433</th>\n",
       "      <td>32.430</td>\n",
       "      <td>3.940</td>\n",
       "      <td>-1.079855</td>\n",
       "      <td>0.536193</td>\n",
       "      <td>-1.681063</td>\n",
       "      <td>-1.682500</td>\n",
       "      <td>-1.602569</td>\n",
       "      <td>-1.663134</td>\n",
       "      <td>-0.537830</td>\n",
       "      <td>-0.396731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43586</th>\n",
       "      <td>13.573</td>\n",
       "      <td>21.040</td>\n",
       "      <td>-1.052372</td>\n",
       "      <td>0.536193</td>\n",
       "      <td>-0.752005</td>\n",
       "      <td>-0.800059</td>\n",
       "      <td>-0.871346</td>\n",
       "      <td>-0.807690</td>\n",
       "      <td>0.135828</td>\n",
       "      <td>-0.761666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43299</th>\n",
       "      <td>34.350</td>\n",
       "      <td>2.677</td>\n",
       "      <td>-0.667608</td>\n",
       "      <td>0.536193</td>\n",
       "      <td>-1.094289</td>\n",
       "      <td>-1.139459</td>\n",
       "      <td>-1.035498</td>\n",
       "      <td>-1.130499</td>\n",
       "      <td>0.001096</td>\n",
       "      <td>0.059438</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Ref. O3(ppb)  Ref. NO2(ppb)   Temp(C)     RH(%)  no2op1(mV)  \\\n",
       "2340         22.223         16.407  0.239336 -1.982867    1.415799   \n",
       "19549        19.247         11.387 -0.008013  0.536193    0.274850   \n",
       "57433        32.430          3.940 -1.079855  0.536193   -1.681063   \n",
       "43586        13.573         21.040 -1.052372  0.536193   -0.752005   \n",
       "43299        34.350          2.677 -0.667608  0.536193   -1.094289   \n",
       "\n",
       "       no2op2(mV)  o3op1(mV)  o3op2(mV)  no2op1 - no2op2(mV)  \\\n",
       "2340     1.372104   1.322321   1.403552             0.809485   \n",
       "19549    0.302993   0.143411   0.306001            -0.133636   \n",
       "57433   -1.682500  -1.602569  -1.663134            -0.537830   \n",
       "43586   -0.800059  -0.871346  -0.807690             0.135828   \n",
       "43299   -1.139459  -1.035498  -1.130499             0.001096   \n",
       "\n",
       "       o3op1 - o3op2(mV)  \n",
       "2340            0.150671  \n",
       "19549          -0.852900  \n",
       "57433          -0.396731  \n",
       "43586          -0.761666  \n",
       "43299           0.059438  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Displaying some top rows of the training data for the dataset DD1(Jun) for split 0  \n",
    "datasets_norm[0]['DD1(Jun).csv']['train'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train models for various calibration algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 0\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 1\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 2\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 3\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 4\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 5\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 6\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 7\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 8\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 9\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "LS LS(MIN) LASSO RT KRR NYS NW(ML) KNN KNN-D KNN(ML) KNN-D(ML) \n"
     ]
    }
   ],
   "source": [
    "algoList = [ 'LS', 'LS(MIN)', 'LASSO', 'RT', 'KRR', 'NYS', 'NW(ML)', 'KNN', 'KNN-D', 'KNN(ML)', 'KNN-D(ML)' ]\n",
    "\n",
    "p.doExperiment( g1, algoList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate MAE, RMSE, MAPE(%), R2 values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.gen_errorMetrics( g1, algoList, datasets, datasets_norm )\n",
    "\n",
    "# Generating mean and std of error metrics over all the splits\n",
    "p.gen_avgErrorMetrics( g1, algoList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Results for KNN-D(ML) for the DD1(Jun) and DD1(Oct) datasets corresponding to Table 4 in the accompanying paper\n",
    "algo = 'KNN-D(ML)'\n",
    "\n",
    "res_o3 = pd.read_csv( g1.errorMetricsPath + algo + '_mean_std_' + 'o3' + '.csv', index_col = 0 )\n",
    "res_no2 = pd.read_csv( g1.errorMetricsPath + algo + '_mean_std_' + 'no2' + '.csv', index_col = 0 )\n",
    "\n",
    "res_t4 = pd.merge(res_o3, res_no2, left_index = True, right_index = True, suffixes=('_O3', '_NO2') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate win matrices and rank all algorithms for each dataset\n",
    "Please note that the ranking and win-matrix results included in this notebook cannot reproduce those in the accompanying paper (Table 3 and Table 5). This is because results in the paper are averaged over 12 datasets whereas those in the notebook use only 2 of those datasets, namely DD1(Jun) and DD1(Oct), which are provided as a part of this repository in the directory [data/](data/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.gen_winMatrix( g1, algoList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Win matrix corresponding to table 3 in the paper (although does not reproduce those results)\n",
    "winMatrix_o3 = pd.read_csv( g1.winMatrixPath + 'o3' + '.csv', index_col = 0 )\n",
    "winMatrix_no2 = pd.read_csv( g1.winMatrixPath + 'no2' + '.csv', index_col = 0 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiments with Alphasense calibration models\n",
    "In our experiments, we found the Alphasense calibration models to perform better with normalized data. However, for sake of completeness, we present here results on both normalized and unnormalized data.\n",
    "\n",
    "**In particular, the results with normalized data in part (a) below reproduce portions of Table S2 in the supplement to the accompanying paper w.r.t the dataset DD1(Jun)**. However, note that Table S2 in the paper also reports results on the dataset DM2(Oct) which are not reproduced below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (a) With normalised data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 0\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 1\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 2\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 3\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 4\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 5\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 6\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 7\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 8\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 9\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n"
     ]
    }
   ],
   "source": [
    "algoList = [ 'AS1', 'AS2', 'AS3', 'AS4' ]\n",
    "\n",
    "p.doExperiment( g1, algoList, datasets, datasets_norm, ASNorm = True )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.gen_errorMetrics( g1, algoList, datasets, datasets_norm )\n",
    "\n",
    "# Generating mean and std of error metrics over all the splits\n",
    "p.gen_avgErrorMetrics( g1, algoList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for AS3 with normalised data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MAE_O3</th>\n",
       "      <th>RMSE_O3</th>\n",
       "      <th>MAPE(%)_O3</th>\n",
       "      <th>R2_O3</th>\n",
       "      <th>MAE_NO2</th>\n",
       "      <th>RMSE_NO2</th>\n",
       "      <th>MAPE(%)_NO2</th>\n",
       "      <th>R2_NO2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>DD1(Jun).csv</th>\n",
       "      <td>251.78±0.27</td>\n",
       "      <td>253.56±0.27</td>\n",
       "      <td>1341.91±120.93</td>\n",
       "      <td>-178.683±3.94</td>\n",
       "      <td>107.7±0.11</td>\n",
       "      <td>108.24±0.12</td>\n",
       "      <td>1954.6±51.77</td>\n",
       "      <td>-95.544±3.504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DD1(Oct).csv</th>\n",
       "      <td>260.39±0.36</td>\n",
       "      <td>261.42±0.34</td>\n",
       "      <td>1473.1±28.05</td>\n",
       "      <td>-58.621±0.642</td>\n",
       "      <td>124.01±0.32</td>\n",
       "      <td>126.08±0.36</td>\n",
       "      <td>780.62±9.84</td>\n",
       "      <td>-28.252±0.678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   MAE_O3      RMSE_O3      MAPE(%)_O3          R2_O3  \\\n",
       "DD1(Jun).csv  251.78±0.27  253.56±0.27  1341.91±120.93  -178.683±3.94   \n",
       "DD1(Oct).csv  260.39±0.36  261.42±0.34    1473.1±28.05  -58.621±0.642   \n",
       "\n",
       "                  MAE_NO2     RMSE_NO2   MAPE(%)_NO2         R2_NO2  \n",
       "DD1(Jun).csv   107.7±0.11  108.24±0.12  1954.6±51.77  -95.544±3.504  \n",
       "DD1(Oct).csv  124.01±0.32  126.08±0.36   780.62±9.84  -28.252±0.678  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Results for AS3 with normalised data') \n",
    "algo = 'AS3'\n",
    "\n",
    "res_o3 = pd.read_csv( g1.errorMetricsPath + algo + '_mean_std_' + 'o3' + '.csv', index_col = 0 )\n",
    "res_no2 = pd.read_csv( g1.errorMetricsPath + algo + '_mean_std_' + 'no2' + '.csv', index_col = 0 )\n",
    "\n",
    "res = pd.merge(res_o3, res_no2, left_index = True, right_index = True, suffixes=('_O3', '_NO2') )\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (b) With un-normalised data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 0\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 1\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 2\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 3\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 4\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 5\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 6\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 7\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 8\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 9\n",
      "\n",
      "Dataset: DD1(Jun).csv\n",
      "AS1 AS2 AS3 AS4 \n",
      "\n",
      "Dataset: DD1(Oct).csv\n",
      "AS1 AS2 AS3 AS4 \n"
     ]
    }
   ],
   "source": [
    "algoList = [ 'AS1', 'AS2', 'AS3', 'AS4' ]\n",
    "\n",
    "p.doExperiment( g1, algoList, datasets, datasets_norm, ASNorm = False )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.gen_errorMetrics( g1, algoList, datasets, datasets_norm )\n",
    "\n",
    "# Generating mean and std of error metrics over all the splits\n",
    "p.gen_avgErrorMetrics( g1, algoList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for AS3 with un-normalised data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MAE_O3</th>\n",
       "      <th>RMSE_O3</th>\n",
       "      <th>MAPE(%)_O3</th>\n",
       "      <th>R2_O3</th>\n",
       "      <th>MAE_NO2</th>\n",
       "      <th>RMSE_NO2</th>\n",
       "      <th>MAPE(%)_NO2</th>\n",
       "      <th>R2_NO2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>DD1(Jun).csv</th>\n",
       "      <td>1044.68±8.31</td>\n",
       "      <td>1416.45±8.9</td>\n",
       "      <td>5442.92±955.24</td>\n",
       "      <td>-5606.108±130.222</td>\n",
       "      <td>313.55±1.98</td>\n",
       "      <td>388.29±2.15</td>\n",
       "      <td>6643.62±300.03</td>\n",
       "      <td>-1241.3±42.954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DD1(Oct).csv</th>\n",
       "      <td>429.65±5.51</td>\n",
       "      <td>610.59±9.39</td>\n",
       "      <td>1828.26±31.53</td>\n",
       "      <td>-324.269±9.118</td>\n",
       "      <td>151.82±1.29</td>\n",
       "      <td>184.28±2.08</td>\n",
       "      <td>1413.03±40.41</td>\n",
       "      <td>-61.517±2.521</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    MAE_O3      RMSE_O3      MAPE(%)_O3              R2_O3  \\\n",
       "DD1(Jun).csv  1044.68±8.31  1416.45±8.9  5442.92±955.24  -5606.108±130.222   \n",
       "DD1(Oct).csv   429.65±5.51  610.59±9.39   1828.26±31.53     -324.269±9.118   \n",
       "\n",
       "                  MAE_NO2     RMSE_NO2     MAPE(%)_NO2          R2_NO2  \n",
       "DD1(Jun).csv  313.55±1.98  388.29±2.15  6643.62±300.03  -1241.3±42.954  \n",
       "DD1(Oct).csv  151.82±1.29  184.28±2.08   1413.03±40.41   -61.517±2.521  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Results for AS3 with un-normalised data')\n",
    "algo = 'AS3'\n",
    "\n",
    "res_o3 = pd.read_csv( g1.errorMetricsPath + algo + '_mean_std_' + 'o3' + '.csv', index_col = 0 )\n",
    "res_no2 = pd.read_csv( g1.errorMetricsPath + algo + '_mean_std_' + 'no2' + '.csv', index_col = 0 )\n",
    "\n",
    "res = pd.merge(res_o3, res_no2, left_index = True, right_index = True, suffixes=('_O3', '_NO2') )\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment 2: calibration results on derived datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate temporally averaged datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gen_derivedDataset as dd\n",
    "\n",
    "# List of different (rolling) window sizes over which averaging is to be done\n",
    "windowList = [ 5, 15, 30, 60 ]\n",
    "\n",
    "datasetList = [ 'DD1(Jun).csv' ]\n",
    "\n",
    "for dataset in datasetList:\n",
    "    for window in windowList:\n",
    "        dd.avg_dataset( g1, dataset, window = window, offset = 0 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate sub-sampled datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "for dataset in datasetList:\n",
    "    dd.small_dataset( g1, dataset )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate aggregated datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasetList = [ 'DD1' ]\n",
    "\n",
    "for dataset in datasetList:\n",
    "    dd.agg_dataset( g1, dataset )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "g2 = globals.My_Globals()\n",
    "g2.datasetPath = g2.derivedDatasetPath\n",
    "g2.datasetList = os.listdir( g2.datasetPath )\n",
    "g2.datasetList.sort()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load a dataset and perform preprocessing\n",
    "**Note**: The calibration algorithms rely on train-test splits to obtain an unbiased estimate of the performance of various calibration algorithms. If you wish to reproduce the results in the paper, please use the train-test supplied for the 2 datasets DD1(Jun) and DD1(Oct) in the directory [perm/](perm/).\n",
    "\n",
    "To do so, please use ```new_permutations = False``` below. Using ```new_permutations = True``` instead, allows fresh train-test splits to be created which will then overwrite the supplied train-test splits. However, minor differences are to be expected in the results if fresh splits are used to conduct the experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets, datasets_norm = p.load_data( g2, new_permutations = False )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ref. O3(ppb)</th>\n",
       "      <th>Ref. NO2(ppb)</th>\n",
       "      <th>Temp(C)</th>\n",
       "      <th>RH(%)</th>\n",
       "      <th>no2op1(mV)</th>\n",
       "      <th>no2op2(mV)</th>\n",
       "      <th>o3op1(mV)</th>\n",
       "      <th>o3op2(mV)</th>\n",
       "      <th>no2op1 - no2op2(mV)</th>\n",
       "      <th>o3op1 - o3op2(mV)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3429</th>\n",
       "      <td>33.6262</td>\n",
       "      <td>12.357400</td>\n",
       "      <td>0.188662</td>\n",
       "      <td>0.533762</td>\n",
       "      <td>0.070043</td>\n",
       "      <td>0.034459</td>\n",
       "      <td>0.062412</td>\n",
       "      <td>0.035154</td>\n",
       "      <td>0.335394</td>\n",
       "      <td>0.188905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8633</th>\n",
       "      <td>37.5220</td>\n",
       "      <td>2.450400</td>\n",
       "      <td>-0.285480</td>\n",
       "      <td>0.533762</td>\n",
       "      <td>-1.016459</td>\n",
       "      <td>-1.006539</td>\n",
       "      <td>-0.968380</td>\n",
       "      <td>-1.011073</td>\n",
       "      <td>-0.435960</td>\n",
       "      <td>-0.206644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>967</th>\n",
       "      <td>32.6926</td>\n",
       "      <td>12.741800</td>\n",
       "      <td>1.368504</td>\n",
       "      <td>-2.457248</td>\n",
       "      <td>1.365363</td>\n",
       "      <td>1.417936</td>\n",
       "      <td>1.361328</td>\n",
       "      <td>1.404907</td>\n",
       "      <td>0.009052</td>\n",
       "      <td>0.386680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7704</th>\n",
       "      <td>34.5014</td>\n",
       "      <td>9.748000</td>\n",
       "      <td>-0.384719</td>\n",
       "      <td>0.533762</td>\n",
       "      <td>0.853108</td>\n",
       "      <td>0.939822</td>\n",
       "      <td>0.762515</td>\n",
       "      <td>0.968711</td>\n",
       "      <td>-0.465627</td>\n",
       "      <td>-0.847058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5348</th>\n",
       "      <td>19.4820</td>\n",
       "      <td>6.595333</td>\n",
       "      <td>-1.228250</td>\n",
       "      <td>0.533762</td>\n",
       "      <td>0.030890</td>\n",
       "      <td>-0.006231</td>\n",
       "      <td>-0.092505</td>\n",
       "      <td>0.009401</td>\n",
       "      <td>0.335394</td>\n",
       "      <td>-0.639865</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Ref. O3(ppb)  Ref. NO2(ppb)   Temp(C)     RH(%)  no2op1(mV)  no2op2(mV)  \\\n",
       "3429       33.6262      12.357400  0.188662  0.533762    0.070043    0.034459   \n",
       "8633       37.5220       2.450400 -0.285480  0.533762   -1.016459   -1.006539   \n",
       "967        32.6926      12.741800  1.368504 -2.457248    1.365363    1.417936   \n",
       "7704       34.5014       9.748000 -0.384719  0.533762    0.853108    0.939822   \n",
       "5348       19.4820       6.595333 -1.228250  0.533762    0.030890   -0.006231   \n",
       "\n",
       "      o3op1(mV)  o3op2(mV)  no2op1 - no2op2(mV)  o3op1 - o3op2(mV)  \n",
       "3429   0.062412   0.035154             0.335394           0.188905  \n",
       "8633  -0.968380  -1.011073            -0.435960          -0.206644  \n",
       "967    1.361328   1.404907             0.009052           0.386680  \n",
       "7704   0.762515   0.968711            -0.465627          -0.847058  \n",
       "5348  -0.092505   0.009401             0.335394          -0.639865  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Displaying some top rows of the training data for the dataset DD1(Jun) for split 0  \n",
    "datasets_norm[0]['DD1(Jun)-AVG5.csv']['train'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train models for various calibration algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 0\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 1\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 2\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 3\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 4\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 5\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 6\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 7\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 8\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Training for split: 9\n",
      "\n",
      "Dataset: DD1(Jun)-AVG15.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG30.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG5.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-AVG60.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun)-SMALL.csv\n",
      "LS NW(ML) KNN-D(ML) \n",
      "\n",
      "Dataset: DD1(Jun-Oct).csv\n",
      "LS NW(ML) KNN-D(ML) \n"
     ]
    }
   ],
   "source": [
    "algoList = [ 'LS', 'NW(ML)', 'KNN-D(ML)' ]\n",
    "\n",
    "p.doExperiment( g2, algoList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate MAE, RMSE, MAPE(%), R2 values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "g2.errorMetricsPath = g2.derivedErrorMetricsPath\n",
    "p.gen_errorMetrics( g2, algoList, datasets, datasets_norm )\n",
    "\n",
    "# Generating mean and std of error metrics over all the splits\n",
    "p.gen_avgErrorMetrics( g2, algoList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Results for KNN-D(ML) and LS for O3 corresponding to Table 6 in the paper\n",
    "algo1, algo2 = 'KNN-D(ML)', 'LS'\n",
    "gas = 'o3'\n",
    "\n",
    "res1 = pd.read_csv( g2.derivedErrorMetricsPath + algo1 + '_mean_std_' + gas + '.csv', index_col = 0 )\n",
    "res2 = pd.read_csv( g2.derivedErrorMetricsPath + algo2 + '_mean_std_' + gas + '.csv', index_col = 0 )\n",
    "res_o3_t6 = pd.merge(res1, res2, left_index = True, right_index = True, suffixes=('_KNN-D(ML)', '_LS') )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Results for KNN-D(ML) and LS for NO2 corresponding to Table 6 in the paper\n",
    "algo1, algo2 = 'KNN-D(ML)', 'LS'\n",
    "gas = 'no2'\n",
    "\n",
    "res1 = pd.read_csv( g2.derivedErrorMetricsPath + algo1 + '_mean_std_' + gas + '.csv', index_col = 0 )\n",
    "res2 = pd.read_csv( g2.derivedErrorMetricsPath + algo2 + '_mean_std_' + gas + '.csv', index_col = 0 )\n",
    "res_no2_t6 = pd.merge(res1, res2, left_index = True, right_index = True, suffixes=('_KNN-D(ML)', '_LS') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment 3: Transfer experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialising Global Variables. For details, refer to globals.py\n",
    "g3 = globals.My_Globals()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load a dataset and perform preprocessing\n",
    "**Note**: The calibration algorithms rely on train-test splits to obtain an unbiased estimate of the performance of various calibration algorithms. If you wish to reproduce the results in the paper, please use the train-test supplied for the 2 datasets DD1(Jun) and DD1(Oct) in the directory [perm/](perm/).\n",
    "\n",
    "To do so, please use ```new_permutations = False``` below. Using ```new_permutations = True``` instead, allows fresh train-test splits to be created which will then overwrite the supplied train-test splits. However, minor differences are to be expected in the results if fresh splits are used to conduct the experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets, datasets_norm = p.load_data( g3, new_permutations = False )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ref. O3(ppb)</th>\n",
       "      <th>Ref. NO2(ppb)</th>\n",
       "      <th>Temp(C)</th>\n",
       "      <th>RH(%)</th>\n",
       "      <th>no2op1(mV)</th>\n",
       "      <th>no2op2(mV)</th>\n",
       "      <th>o3op1(mV)</th>\n",
       "      <th>o3op2(mV)</th>\n",
       "      <th>no2op1 - no2op2(mV)</th>\n",
       "      <th>o3op1 - o3op2(mV)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14385</th>\n",
       "      <td>101.743</td>\n",
       "      <td>9.919</td>\n",
       "      <td>35.5</td>\n",
       "      <td>99.9</td>\n",
       "      <td>134.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>184.0</td>\n",
       "      <td>159.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30727</th>\n",
       "      <td>9.865</td>\n",
       "      <td>38.000</td>\n",
       "      <td>27.6</td>\n",
       "      <td>99.9</td>\n",
       "      <td>84.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45673</th>\n",
       "      <td>21.437</td>\n",
       "      <td>2.306</td>\n",
       "      <td>28.0</td>\n",
       "      <td>99.9</td>\n",
       "      <td>158.0</td>\n",
       "      <td>153.0</td>\n",
       "      <td>167.0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42475</th>\n",
       "      <td>21.127</td>\n",
       "      <td>5.662</td>\n",
       "      <td>26.7</td>\n",
       "      <td>99.9</td>\n",
       "      <td>89.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12073</th>\n",
       "      <td>19.567</td>\n",
       "      <td>8.581</td>\n",
       "      <td>28.1</td>\n",
       "      <td>99.9</td>\n",
       "      <td>192.0</td>\n",
       "      <td>176.0</td>\n",
       "      <td>209.0</td>\n",
       "      <td>201.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Ref. O3(ppb)  Ref. NO2(ppb)  Temp(C)  RH(%)  no2op1(mV)  no2op2(mV)  \\\n",
       "14385       101.743          9.919     35.5   99.9       134.0       134.0   \n",
       "30727         9.865         38.000     27.6   99.9        84.0        61.0   \n",
       "45673        21.437          2.306     28.0   99.9       158.0       153.0   \n",
       "42475        21.127          5.662     26.7   99.9        89.0        84.0   \n",
       "12073        19.567          8.581     28.1   99.9       192.0       176.0   \n",
       "\n",
       "       o3op1(mV)  o3op2(mV)  no2op1 - no2op2(mV)  o3op1 - o3op2(mV)  \n",
       "14385      184.0      159.0                  0.0               25.0  \n",
       "30727       95.0       83.0                 23.0               12.0  \n",
       "45673      167.0      180.0                  5.0              -13.0  \n",
       "42475       99.0      106.0                  5.0               -7.0  \n",
       "12073      209.0      201.0                 16.0                8.0  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Displaying some top rows of the testing data for the dataset DD1(Jun) for split 0  \n",
    "datasets[0]['DD1(Jun).csv']['test'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test models by attempting to transfer them to a different dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 0\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 1\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 2\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 3\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 4\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 5\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 6\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 7\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 8\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n",
      "\n",
      "--------------------------------------------------------------\n",
      "Testing for split: 9\n",
      "\n",
      "trainDataset: DD1(Jun).csv testDataset: DD1(Oct).csv\n",
      "LS KNN-D(ML) \n"
     ]
    }
   ],
   "source": [
    "algoList = ['LS','KNN-D(ML)']\n",
    "datasetTrainList = ['DD1(Jun).csv']\n",
    "datasetTestList  = ['DD1(Oct).csv']\n",
    "datasetList = [ datasetTrain[:-4] + ' -> ' + datasetTest[:-4] for datasetTrain, datasetTest in zip( datasetTrainList, datasetTestList ) ]\n",
    "\n",
    "p.transferTesting( g3, algoList, datasetTrainList, datasetTestList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate MAE, RMSE, MAPE(%), R2 values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.gen_transferErrorMetrics( g3, algoList, datasetTrainList, datasetTestList, datasetList, datasets, datasets_norm )\n",
    "\n",
    "g3.errorMetricsPath = g3.transferErrorMetricsPath\n",
    "g3.datasetList = datasetList\n",
    "# Generating mean and std of error metrics over all the splits\n",
    "p.gen_avgErrorMetrics( g3, algoList, datasets, datasets_norm )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Results for KNN-D(ML) and LS for O3 corresponding to Table 6 in the paper\n",
    "algo1, algo2 = 'KNN-D(ML)', 'LS'\n",
    "gas = 'o3'\n",
    "\n",
    "res1 = pd.read_csv( g3.transferErrorMetricsPath + algo1 + '_mean_std_' + gas + '.csv', index_col = 0 )\n",
    "res2 = pd.read_csv( g3.transferErrorMetricsPath + algo2 + '_mean_std_' + gas + '.csv', index_col = 0 )\n",
    "res = pd.merge(res1, res2, left_index = True, right_index = True, suffixes=('_KNN-D(ML)', '_LS') )\n",
    "res_o3_t6 = res.append(res_o3_t6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Results for KNN-D(ML) and LS for NO2 corresponding to Table 6 in the paper\n",
    "algo1, algo2 = 'KNN-D(ML)', 'LS'\n",
    "gas = 'no2'\n",
    "\n",
    "res1 = pd.read_csv( g3.transferErrorMetricsPath + algo1 + '_mean_std_' + gas + '.csv', index_col = 0 )\n",
    "res2 = pd.read_csv( g3.transferErrorMetricsPath + algo2 + '_mean_std_' + gas + '.csv', index_col = 0 )\n",
    "res = pd.merge(res1, res2, left_index = True, right_index = True, suffixes=('_KNN-D(ML)', '_LS') )\n",
    "res_no2_t6 = res.append(res_no2_t6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. MAE and R2 for KNN-D(ML) averaged over 10 splits.\n",
    "**This reproduces portions of Table 4 in the accompanying paper w.r.t the dataset DD1(Jun)**. Note that Table 4 in the paper also reports results on the MM5(Jun) and (Oct) datasets which are not reproduced below. Also, Table 4 in the paper does not report MAE values which are reported below.\n",
    "\n",
    "Upon executing the code block below, the following table should be printed as output if code execution has been performed correctly.\n",
    "#### <div align=\"center\">Expected Results for KNN-D(ML)</div>\n",
    "\n",
    "||MAE_O3|R2_O3|MAE_NO2|R2_NO2|\n",
    "|---|---|---|---|---|\n",
    "|__DD1(Jun).csv__|3.52 ± 0.03|0.923 ± 0.003|2.67 ± 0.05|0.819 ± 0.015|\n",
    "|__DD1(Oct).csv__|2.1 ± 0.06|0.99 ± 0.001|2.16 ± 0.05|0.977 ± 0.002|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MAE_O3</th>\n",
       "      <th>R2_O3</th>\n",
       "      <th>MAE_NO2</th>\n",
       "      <th>R2_NO2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>DD1(Jun).csv</th>\n",
       "      <td>3.52±0.03</td>\n",
       "      <td>0.923±0.003</td>\n",
       "      <td>2.67±0.05</td>\n",
       "      <td>0.819±0.015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DD1(Oct).csv</th>\n",
       "      <td>2.1±0.06</td>\n",
       "      <td>0.99±0.001</td>\n",
       "      <td>2.16±0.05</td>\n",
       "      <td>0.977±0.002</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 MAE_O3        R2_O3    MAE_NO2       R2_NO2\n",
       "DD1(Jun).csv  3.52±0.03  0.923±0.003  2.67±0.05  0.819±0.015\n",
       "DD1(Oct).csv   2.1±0.06   0.99±0.001  2.16±0.05  0.977±0.002"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_t4[['MAE_O3', 'R2_O3', 'MAE_NO2', 'R2_NO2']].loc[['DD1(Jun).csv', 'DD1(Oct).csv']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Win Matrix for O<sub>3</sub> averaged over 2 datasets\n",
    "**Please note that although this corresponds to the experiment reported by Table 3 in the accompanying paper, those results are averaged over 12 datasets and cannot be reproduced by the results below that are averaged over only 2 datasets.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LS</th>\n",
       "      <th>RT</th>\n",
       "      <th>KRR</th>\n",
       "      <th>NW(ML)</th>\n",
       "      <th>KNN-D(ML)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LS</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RT</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KRR</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NW(ML)</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KNN-D(ML)</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            LS   RT   KRR  NW(ML)  KNN-D(ML)\n",
       "LS         0.0  0.0  0.00    0.00        0.0\n",
       "RT         1.0  0.0  0.45    0.00        0.0\n",
       "KRR        1.0  0.5  0.00    0.05        0.0\n",
       "NW(ML)     1.0  1.0  0.80    0.00        0.0\n",
       "KNN-D(ML)  1.0  1.0  1.00    1.00        0.0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "winMatrix_o3[['LS', 'RT', 'KRR', 'NW(ML)', 'KNN-D(ML)']].loc[['LS', 'RT', 'KRR', 'NW(ML)', 'KNN-D(ML)']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Win Matrix for NO<sub>2</sub> averaged over 2 datasets\n",
    "**Please note that although this corresponds to the experiment reported by Table 3 in the accompanying paper, those results are averaged over 12 datasets and cannot be reproduced by the results below that are averaged over only 2 datasets.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LS</th>\n",
       "      <th>RT</th>\n",
       "      <th>KRR</th>\n",
       "      <th>NW(ML)</th>\n",
       "      <th>KNN-D(ML)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LS</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RT</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KRR</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NW(ML)</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KNN-D(ML)</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            LS   RT  KRR  NW(ML)  KNN-D(ML)\n",
       "LS         0.0  0.0  0.0    0.00        0.0\n",
       "RT         1.0  0.0  1.0    0.35        0.0\n",
       "KRR        1.0  0.0  0.0    0.00        0.0\n",
       "NW(ML)     1.0  0.5  1.0    0.00        0.0\n",
       "KNN-D(ML)  1.0  1.0  1.0    1.00        0.0"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "winMatrix_no2[['LS', 'RT', 'KRR', 'NW(ML)', 'KNN-D(ML)']].loc[['LS', 'RT', 'KRR', 'NW(ML)', 'KNN-D(ML)']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. MAE and R2 for O<sub>3</sub> in transfer experiments\n",
    "**This reproduces portions of Table 6 in the accompanying paper w.r.t the transfer experiments DD1(Jun) -> (Oct) and the dataset DD1(Jun-Oct) for the gas O3**. However, note that Table 6 in the paper also reports results on several other transfers e.g. MM5(Oct) -> (Jun) which are not reproduced below.\n",
    "\n",
    "Upon executing the code block below, the following table should be printed as output if code execution has been performed correctly.\n",
    "#### <div align=\"center\">Expected Results for KNN-D(ML) and LS for O<sub>3</sub></div>\n",
    "\n",
    "||MAE_KNN-D(ML)|R2_KNN-D(ML)|MAE_LS|R2_LS|\n",
    "|---|---|---|---|---|\n",
    "|__DD1(Jun) -> DD1(Oct)__|21.7 ± 2.57|0.193 ± 0.188|12.88 ± 0.17|0.729 ± 0.004|\n",
    "|__DD1(Jun-Oct).csv__|3.29 ± 0.07|0.956 ± 0.002|7.17 ± 0.02|0.842 ± 0.002|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MAE_KNN-D(ML)</th>\n",
       "      <th>R2_KNN-D(ML)</th>\n",
       "      <th>MAE_LS</th>\n",
       "      <th>R2_LS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>DD1(Jun) -&gt; DD1(Oct)</th>\n",
       "      <td>21.7±2.57</td>\n",
       "      <td>0.193±0.188</td>\n",
       "      <td>12.88±0.17</td>\n",
       "      <td>0.729±0.004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DD1(Jun-Oct).csv</th>\n",
       "      <td>3.29±0.07</td>\n",
       "      <td>0.956±0.002</td>\n",
       "      <td>7.17±0.02</td>\n",
       "      <td>0.842±0.002</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     MAE_KNN-D(ML) R2_KNN-D(ML)      MAE_LS        R2_LS\n",
       "DD1(Jun) -> DD1(Oct)     21.7±2.57  0.193±0.188  12.88±0.17  0.729±0.004\n",
       "DD1(Jun-Oct).csv         3.29±0.07  0.956±0.002   7.17±0.02  0.842±0.002"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_o3_t6[['MAE_KNN-D(ML)', 'R2_KNN-D(ML)', 'MAE_LS', 'R2_LS']].loc[['DD1(Jun) -> DD1(Oct)','DD1(Jun-Oct).csv']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. MAE and R2 for NO<sub>2</sub> in transfer experiments\n",
    "**This reproduces portions of Table 6 in the accompanying paper w.r.t the transfer experiments DD1(Jun) -> (Oct) and the dataset DD1(Jun-Oct) for the gas NO2**. However, note that Table 6 in the paper also reports results on several other transfers e.g. MM5(Oct) -> (Jun) which are not reproduced below.\n",
    "\n",
    "Upon executing the code block below, the following table should be printed as output if code execution has been performed correctly.\n",
    "#### <div align=\"center\">Expected Results for KNN-D(ML) and LS for NO<sub>2</sub></div>\n",
    "\n",
    "||MAE_KNN-D(ML)|R2_KNN-D(ML)|MAE_LS|R2_LS|\n",
    "|---|---|---|---|---|\n",
    "|__DD1(Jun) -> DD1(Oct)__|21.86 ± 2.07|-0.642 ± 0.268|12.73 ± 0.2|0.219 ± 0.007| \n",
    "|__DD1(Jun-Oct).csv__|2.62 ± 0.07|0.924 ± 0.003|8.16 ± 0.03|0.482 ± 0.007| "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MAE_KNN-D(ML)</th>\n",
       "      <th>R2_KNN-D(ML)</th>\n",
       "      <th>MAE_LS</th>\n",
       "      <th>R2_LS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>DD1(Jun) -&gt; DD1(Oct)</th>\n",
       "      <td>21.86±2.07</td>\n",
       "      <td>-0.642±0.268</td>\n",
       "      <td>12.73±0.2</td>\n",
       "      <td>0.219±0.007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DD1(Jun-Oct).csv</th>\n",
       "      <td>2.62±0.07</td>\n",
       "      <td>0.924±0.003</td>\n",
       "      <td>8.16±0.03</td>\n",
       "      <td>0.482±0.007</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     MAE_KNN-D(ML)  R2_KNN-D(ML)     MAE_LS        R2_LS\n",
       "DD1(Jun) -> DD1(Oct)    21.86±2.07  -0.642±0.268  12.73±0.2  0.219±0.007\n",
       "DD1(Jun-Oct).csv         2.62±0.07   0.924±0.003  8.16±0.03  0.482±0.007"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_no2_t6[['MAE_KNN-D(ML)', 'R2_KNN-D(ML)', 'MAE_LS', 'R2_LS']].loc[['DD1(Jun) -> DD1(Oct)','DD1(Jun-Oct).csv']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "author": {
   "email": "anagal@cse.iitk.ac.in, purushot@cse.iitk.ac.in",
   "institution": "IIT Kanpur",
   "name": "Ayush Nagal, Purushottam Kar"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}