Final_Report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import And Export Analysis Through Sea and Air\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "* [Introduction](#Introduction)\n",
    "* [Data Preprocessing](#Data-Preprocessing)\n",
    "* [Data Processing](#Data-Processing)\n",
    "* [Model Selection](#Model-Selection)\n",
    "* [Clustering](#Clustering)\n",
    "* [Policies](#Policies)\n",
    "* [Conclusion](#Conclusion)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This report was made to summarize the results of the project for Knowledge Engineering Lab (CS307).\n",
    "\n",
    "The task was to download a novel Indian government dataset, work on it to get a tangible useful result by the methods of data mining and analysis\n",
    "\n",
    "The dataset is taken from [data.gov.in](data.gov.in) on the **Import and Export of Commodities through Sea and Air in the State of Tamil Nadu**. That is, the four datasets are, Import Commodities through Sea, Import Commodities through Air, Export Commodities through Sea and Export Commodites through Air."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import folium\n",
    "import matplotlib.cm as cm\n",
    "import matplotlib.colors as colors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing\n",
    "\n",
    "The following steps were taken to process the data-\n",
    "\n",
    "* The dataset had the @ character substituted for null values. This was replaced by zero as zero makes more intuitive sense.\n",
    "* Any Row which had a Total entry was removed. This was done as it did not convey a meaning greater than the sum of its parts.\n",
    "* Empty Row entries were Removed. (They were marked as a 'False' entry in the Commodity name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Processing\n",
    "\n",
    "These were the steps that followed-\n",
    "\n",
    "* The datasets were pivoted with the Name of the Country as the index, the Commodity group as the Columns and the Value of Trade (Rs. in crores) as the Value.\n",
    "* Duplicate Entries due to variation in name of the Countries was fixed by summing the Duplicates. For example: UAE had a duplicate named United Arab Emirates. Their value was summed and UAE was chosen as the entry name.\n",
    "* The Latitude and Longitude of the countries were appended to the datasets for easy plotting of the countries.\n",
    "\n",
    "[Notebook1](Data_Cleaning.ipynb)\n",
    "\n",
    "\n",
    "[Notebook2](Getting_countries_latitude_and_longitude.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Selection\n",
    "\n",
    "* The dataset was trained with different clustering models with a common score (the calinski_harasbasz's score) to find the best training model.\n",
    "* K Means performed best on all of the datasets except for Import commodities by Air which used AffinityPropagation.\n",
    "\n",
    "[Notebook](Clustering_Model_Selection.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clustering \n",
    "\n",
    "* The dataset was then clustered with their respective models.\n",
    "* Their cluster number was then appended to a copy of the dataset and then saved.\n",
    "\n",
    "[Notebook 1](Clustering.ipynb)\n",
    "\n",
    "\n",
    "[Notebook 2](post_analysis/Clusters_data_sets.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Policies \n",
    "\n",
    "* An initial draft of the policies were created looking at the disparities in the volume of goods transfered.\n",
    "* The policies were then filtered and the most important ones (according to volume of trade were selected).\n",
    "\n",
    "[Notebook](post_analysis/Policies.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "The dataset is taken from [data.gov.in](data.gov.in) on the Import and Export of Commodities through Sea and Air was sanity checked, filtered, transformed, clustered and again transformed into policies for use for policy makers\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}