hcds-a2-bias.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A2: Bias in data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Author: Haowen Ni\n",
    "* Date: 11/02/2017\n",
    "  \n",
    "In this assignment, the author:  \n",
    "1. Collects politicans articles by coountry from English Wikipedia dataset \n",
    "2. Collects population by country dataset\n",
    "3. Use a machine learning API called ORES to get the article quality data of articles on political figures from English Wikipedia\n",
    "4. Combines both article quality data and population data together into a dataframe \n",
    "5. Performs an analysis of the differences on the coverage of politicians on Wikipedia and the article quality about politicians between various countries \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aritcle and population data loading and preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import Python packages that we will use in this script\n",
    "import csv\n",
    "import json\n",
    "import pandas\n",
    "import pickle\n",
    "import requests\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first import the English Wikipedia politicians articles dataset and population dataset into this notebook working environment. Both datasets are CSV files. The Wikipedia articles dataset contains the information about the sanitised country name, the unsanitised page title, and the edit ID of the last edit to the page (also called revision id.) The population dataset contains the information about the country name and corresponding population. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import wikipedia dataset\n",
    "\n",
    "wiki_data = []\n",
    "with open('page_data.csv') as csvfile:\n",
    "    reader = csv.reader(csvfile)\n",
    "    for row in reader:\n",
    "        wiki_data.append(row)\n",
    "wiki_data = wiki_data[1:] # Skip the header\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the population data are in string format, so we need to convert them to integers before doing calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import population data\n",
    "\n",
    "population_list = []\n",
    "with open('Population Mid-2015.csv') as csvfile:\n",
    "    reader = csv.reader(csvfile)\n",
    "    next(reader) # Skip table title\n",
    "    next(reader) # Skip emtpy line\n",
    "    next(reader) # Skip table headers\n",
    "    for row in reader:\n",
    "        if len(row) < 5: # Skip empty rows\n",
    "            continue\n",
    "        row[4] = int(row[4].replace(',', ''))\n",
    "        population_list.append(row)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the Wikipedia dataset we just imported, a dictionary is constructed in which the keys are revision ids and the values are tuples of country names and article names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Based on wikipedia dataset, construct a dictionary as following:\n",
    "# Key: revision id\n",
    "# Value: (country name, article name)\n",
    "\n",
    "country_article = {}\n",
    "for line in wiki_data:\n",
    "    country_article[line[2]] = (line[1], line[0])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the population dataset we just imported, a dictionary is constructed in which the keys are country names and the values are population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Based on population data, construct a dictionary as following:\n",
    "# Key: country name\n",
    "# Value: population\n",
    "\n",
    "population_data = {}\n",
    "for line in population_list:\n",
    "    population_data[line[0]] = line[4]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using ORES to get article quality predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to get the article quality predictions, we will use ORES, a machine learning API provided by Wikipedia. We write a function that accepts an revision id and API user header as parameters and returns the article quality prediction. Since multiple revision ids can be passed into ORES API for each call, we split the Wikipedia article set into batches of 50. Notice that not every revision id from Wikipedia datasets that get passed into the API has a article quality prediction. For those articles that cannot be predicted, we just raise a flag and ignore them in the final prediction dataset. The result of this machine learning model is in JSON format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "headers = {'User-Agent' : 'https://github.com/HWNi', 'From' : 'haowen2@uw.edu'}\n",
    "\n",
    "def get_ores_data(revision_ids, headers):\n",
    "    \n",
    "    # Define the endpoint\n",
    "    endpoint = 'https://ores.wikimedia.org/v3/scores/{project}/?models={model}&revids={revids}'\n",
    "    \n",
    "    # Specify the parameters - smushing all the revision IDs together separated by | marks.\n",
    "    # Yes, 'smush' is a technical term, trust me I'm a scientist.\n",
    "    # What do you mean \"but people trusting scientists regularly goes horribly wrong\" who taught you tha- oh.  \n",
    "    params = {'project' : 'enwiki',\n",
    "              'model'   : 'wp10',\n",
    "              'revids'  : '|'.join(str(x) for x in revision_ids)\n",
    "              }\n",
    "    api_call = requests.get(endpoint.format(**params))\n",
    "    response = api_call.json()\n",
    "    return response\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Could not find revision 806811023\n",
      "Could not find revision 807367030\n",
      "Could not find revision 807367166\n",
      "Could not find revision 807484325\n"
     ]
    }
   ],
   "source": [
    "# Gather prediction for each article\n",
    "# It's recommended to batch 50 revisions in each request \n",
    "\n",
    "rev_id_list = [line[2] for line in wiki_data[1:]]\n",
    "\n",
    "prediction_data = {}\n",
    "\n",
    "fifty_ids = []\n",
    "for counter, rev_id in enumerate(rev_id_list):  \n",
    "    if counter % 50 == 0 and counter != 0:\n",
    "        fifty_predictions = get_ores_data(fifty_ids, headers)\n",
    "        fifty_predictions = fifty_predictions['enwiki']['scores'].items()\n",
    "        for key, value in fifty_predictions:            \n",
    "            try: \n",
    "                prediction_data[key] = value['wp10']['score']['prediction']\n",
    "            except KeyError:\n",
    "                print('Could not find revision ' + str(key))\n",
    "        fifty_ids = []\n",
    "    fifty_ids.append(rev_id)\n",
    "\n",
    "# Gather predictions for last 47 revisions \n",
    "fifty_predictions = get_ores_data(fifty_ids, headers)\n",
    "fifty_predictions = fifty_predictions['enwiki']['scores'].items()\n",
    "for key, value in fifty_predictions:            \n",
    "    try: \n",
    "        prediction_data[key] = value['wp10']['score']['prediction']\n",
    "    except KeyError:\n",
    "        print('Could not find revision ' + str(key))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could save the prediction data in local disk by using pickle so that we could reuse the article quality prediction data when we restart the instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Save the prediction data for later use\n",
    "pickle.dump(prediction_data, open('prediction_data.p', 'wb'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the saved prediction data\n",
    "prediction_data = pickle.load(open('prediction_data.p', 'rb'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining the datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we merge both datasets we just collected based on country names into a dataframe. Notice that country names from Wikipedia articles dataset may not match those from population dataset. In those cases, we just ignore those data with country names that do not match. We eventually save the table into a csv file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>article_name</th>\n",
       "      <th>revision_id</th>\n",
       "      <th>article_quality</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Chad</td>\n",
       "      <td>Bir I of Kanem</td>\n",
       "      <td>355319463</td>\n",
       "      <td>Stub</td>\n",
       "      <td>13707000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Zimbabwe</td>\n",
       "      <td>Template:Zimbabwe-politician-stub</td>\n",
       "      <td>391862046</td>\n",
       "      <td>Stub</td>\n",
       "      <td>17354000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Uganda</td>\n",
       "      <td>Template:Uganda-politician-stub</td>\n",
       "      <td>391862070</td>\n",
       "      <td>Stub</td>\n",
       "      <td>40141000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Namibia</td>\n",
       "      <td>Template:Namibia-politician-stub</td>\n",
       "      <td>391862409</td>\n",
       "      <td>Stub</td>\n",
       "      <td>2482100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nigeria</td>\n",
       "      <td>Template:Nigeria-politician-stub</td>\n",
       "      <td>391862819</td>\n",
       "      <td>Stub</td>\n",
       "      <td>181839400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Colombia</td>\n",
       "      <td>Template:Colombia-politician-stub</td>\n",
       "      <td>391863340</td>\n",
       "      <td>Stub</td>\n",
       "      <td>48218000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Chile</td>\n",
       "      <td>Template:Chile-politician-stub</td>\n",
       "      <td>391863361</td>\n",
       "      <td>Stub</td>\n",
       "      <td>18025000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Fiji</td>\n",
       "      <td>Template:Fiji-politician-stub</td>\n",
       "      <td>391863617</td>\n",
       "      <td>Stub</td>\n",
       "      <td>867000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Solomon Islands</td>\n",
       "      <td>Template:Solomons-politician-stub</td>\n",
       "      <td>391863809</td>\n",
       "      <td>Stub</td>\n",
       "      <td>641900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Palestinian Territory</td>\n",
       "      <td>Information Minister of the Palestinian Nation...</td>\n",
       "      <td>393276188</td>\n",
       "      <td>Stub</td>\n",
       "      <td>4481195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Somalia</td>\n",
       "      <td>Template:Somalia-politician-stub</td>\n",
       "      <td>393298432</td>\n",
       "      <td>Stub</td>\n",
       "      <td>11123000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Cambodia</td>\n",
       "      <td>Yos Por</td>\n",
       "      <td>393822005</td>\n",
       "      <td>Stub</td>\n",
       "      <td>15417100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Slovakia</td>\n",
       "      <td>Template:Slovakia-politician-stub</td>\n",
       "      <td>394482629</td>\n",
       "      <td>Stub</td>\n",
       "      <td>5424051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Slovenia</td>\n",
       "      <td>Template:Slovenia-politician-stub</td>\n",
       "      <td>394482891</td>\n",
       "      <td>Stub</td>\n",
       "      <td>2064000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>Template:Afghanistan-politician-stub</td>\n",
       "      <td>394580295</td>\n",
       "      <td>Stub</td>\n",
       "      <td>32247000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Iraq</td>\n",
       "      <td>Template:Iraq-politician-stub</td>\n",
       "      <td>394580630</td>\n",
       "      <td>Stub</td>\n",
       "      <td>37056000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Nepal</td>\n",
       "      <td>Template:Nepal-politician-stub</td>\n",
       "      <td>394580939</td>\n",
       "      <td>Stub</td>\n",
       "      <td>28039000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Sri Lanka</td>\n",
       "      <td>Template:SriLanka-politician-stub</td>\n",
       "      <td>394580993</td>\n",
       "      <td>Stub</td>\n",
       "      <td>20868800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Laos</td>\n",
       "      <td>Template:Laos-politician-stub</td>\n",
       "      <td>394581284</td>\n",
       "      <td>Stub</td>\n",
       "      <td>6903049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Albania</td>\n",
       "      <td>Template:Albania-politician-stub</td>\n",
       "      <td>394581557</td>\n",
       "      <td>Stub</td>\n",
       "      <td>2892000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Costa Rica</td>\n",
       "      <td>Template:CostaRica-politician-stub</td>\n",
       "      <td>394587483</td>\n",
       "      <td>Stub</td>\n",
       "      <td>4832000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Czech Republic</td>\n",
       "      <td>Julius Gregr</td>\n",
       "      <td>395521877</td>\n",
       "      <td>Stub</td>\n",
       "      <td>10551227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Czech Republic</td>\n",
       "      <td>Edvard Gregr</td>\n",
       "      <td>395526568</td>\n",
       "      <td>Stub</td>\n",
       "      <td>10551227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Canada</td>\n",
       "      <td>Robert Douglas Cook</td>\n",
       "      <td>401577829</td>\n",
       "      <td>Stub</td>\n",
       "      <td>35833000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Tunisia</td>\n",
       "      <td>Template:Tunisia-politician-stub</td>\n",
       "      <td>413885084</td>\n",
       "      <td>Stub</td>\n",
       "      <td>11026000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Guatemala</td>\n",
       "      <td>Template:Guatemala-politician-stub</td>\n",
       "      <td>433871129</td>\n",
       "      <td>Stub</td>\n",
       "      <td>16183752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Burkina Faso</td>\n",
       "      <td>Template:BurkinaFaso-politician-stub</td>\n",
       "      <td>433871165</td>\n",
       "      <td>Stub</td>\n",
       "      <td>18450400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Angola</td>\n",
       "      <td>Template:Angola-politician-stub</td>\n",
       "      <td>435008715</td>\n",
       "      <td>Stub</td>\n",
       "      <td>25000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Panama</td>\n",
       "      <td>Template:Panama-politician-stub</td>\n",
       "      <td>437454659</td>\n",
       "      <td>Stub</td>\n",
       "      <td>3980000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Japan</td>\n",
       "      <td>Template:Japan-politician-1980s-stub</td>\n",
       "      <td>437735138</td>\n",
       "      <td>Stub</td>\n",
       "      <td>126866820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45764</th>\n",
       "      <td>Philippines</td>\n",
       "      <td>President of the Philippines</td>\n",
       "      <td>807475001</td>\n",
       "      <td>B</td>\n",
       "      <td>102965300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45765</th>\n",
       "      <td>Spain</td>\n",
       "      <td>Jaime del Burgo Torres</td>\n",
       "      <td>807475042</td>\n",
       "      <td>FA</td>\n",
       "      <td>46368000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45766</th>\n",
       "      <td>Turkey</td>\n",
       "      <td>Tunahan Kuzu</td>\n",
       "      <td>807475476</td>\n",
       "      <td>Start</td>\n",
       "      <td>78215000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45767</th>\n",
       "      <td>Russia</td>\n",
       "      <td>Mark Feygin</td>\n",
       "      <td>807475576</td>\n",
       "      <td>C</td>\n",
       "      <td>144302000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45768</th>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>Bernard Williams</td>\n",
       "      <td>807476248</td>\n",
       "      <td>FA</td>\n",
       "      <td>65092000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45769</th>\n",
       "      <td>Kenya</td>\n",
       "      <td>Alice Muthoni Wahome</td>\n",
       "      <td>807476386</td>\n",
       "      <td>Stub</td>\n",
       "      <td>44306000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45770</th>\n",
       "      <td>Australia</td>\n",
       "      <td>Don Chipp</td>\n",
       "      <td>807476693</td>\n",
       "      <td>C</td>\n",
       "      <td>23888000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45771</th>\n",
       "      <td>Israel</td>\n",
       "      <td>Avi Gabbay</td>\n",
       "      <td>807476738</td>\n",
       "      <td>C</td>\n",
       "      <td>8375384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45772</th>\n",
       "      <td>Australia</td>\n",
       "      <td>Greg Wilton</td>\n",
       "      <td>807477468</td>\n",
       "      <td>Start</td>\n",
       "      <td>23888000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45773</th>\n",
       "      <td>Somalia</td>\n",
       "      <td>Siad Barre</td>\n",
       "      <td>807477894</td>\n",
       "      <td>B</td>\n",
       "      <td>11123000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45774</th>\n",
       "      <td>India</td>\n",
       "      <td>V.P. Deenadayalu Naidu</td>\n",
       "      <td>807478407</td>\n",
       "      <td>Stub</td>\n",
       "      <td>1314097616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45775</th>\n",
       "      <td>Kazakhstan</td>\n",
       "      <td>Nursultan Nazarbayev</td>\n",
       "      <td>807478471</td>\n",
       "      <td>B</td>\n",
       "      <td>17544274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45776</th>\n",
       "      <td>Philippines</td>\n",
       "      <td>Aga Muhlach</td>\n",
       "      <td>807479087</td>\n",
       "      <td>B</td>\n",
       "      <td>102965300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45777</th>\n",
       "      <td>Fiji</td>\n",
       "      <td>Elizabeth II</td>\n",
       "      <td>807479170</td>\n",
       "      <td>FA</td>\n",
       "      <td>867000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45778</th>\n",
       "      <td>Albania</td>\n",
       "      <td>Haki Toska</td>\n",
       "      <td>807479431</td>\n",
       "      <td>Start</td>\n",
       "      <td>2892000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45779</th>\n",
       "      <td>Bosnia-Herzegovina</td>\n",
       "      <td>Hakija Turajlić</td>\n",
       "      <td>807479443</td>\n",
       "      <td>Start</td>\n",
       "      <td>3650114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45780</th>\n",
       "      <td>Philippines</td>\n",
       "      <td>King Gutierrez</td>\n",
       "      <td>807479587</td>\n",
       "      <td>Start</td>\n",
       "      <td>102965300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45781</th>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>Mark Boleat</td>\n",
       "      <td>807479783</td>\n",
       "      <td>Stub</td>\n",
       "      <td>65092000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45782</th>\n",
       "      <td>Canada</td>\n",
       "      <td>Amir Khadir</td>\n",
       "      <td>807479923</td>\n",
       "      <td>C</td>\n",
       "      <td>35833000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45783</th>\n",
       "      <td>India</td>\n",
       "      <td>Mangal Das Pakvasa</td>\n",
       "      <td>807479965</td>\n",
       "      <td>Stub</td>\n",
       "      <td>1314097616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45784</th>\n",
       "      <td>Germany</td>\n",
       "      <td>Hakkı Keskin</td>\n",
       "      <td>807480590</td>\n",
       "      <td>Start</td>\n",
       "      <td>81132000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45785</th>\n",
       "      <td>China</td>\n",
       "      <td>Yang Jiechi</td>\n",
       "      <td>807481066</td>\n",
       "      <td>C</td>\n",
       "      <td>1371920000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45786</th>\n",
       "      <td>China</td>\n",
       "      <td>Li Jiancheng</td>\n",
       "      <td>807481329</td>\n",
       "      <td>C</td>\n",
       "      <td>1371920000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45787</th>\n",
       "      <td>Iran</td>\n",
       "      <td>Mahmoud Ahmadinejad</td>\n",
       "      <td>807481450</td>\n",
       "      <td>B</td>\n",
       "      <td>78483446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45788</th>\n",
       "      <td>Greece</td>\n",
       "      <td>George I of Greece</td>\n",
       "      <td>807481543</td>\n",
       "      <td>FA</td>\n",
       "      <td>11520785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45789</th>\n",
       "      <td>United States</td>\n",
       "      <td>Hal Bidlack</td>\n",
       "      <td>807481636</td>\n",
       "      <td>C</td>\n",
       "      <td>321234172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45790</th>\n",
       "      <td>Gambia</td>\n",
       "      <td>Yahya Jammeh</td>\n",
       "      <td>807482007</td>\n",
       "      <td>GA</td>\n",
       "      <td>2021893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45791</th>\n",
       "      <td>United States</td>\n",
       "      <td>Lucius Fairchild</td>\n",
       "      <td>807483006</td>\n",
       "      <td>C</td>\n",
       "      <td>321234172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45792</th>\n",
       "      <td>Saudi Arabia</td>\n",
       "      <td>Fahd of Saudi Arabia</td>\n",
       "      <td>807483153</td>\n",
       "      <td>GA</td>\n",
       "      <td>31565109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45793</th>\n",
       "      <td>United States</td>\n",
       "      <td>Francis Fessenden</td>\n",
       "      <td>807483270</td>\n",
       "      <td>C</td>\n",
       "      <td>321234172</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>45794 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     country  \\\n",
       "0                       Chad   \n",
       "1                   Zimbabwe   \n",
       "2                     Uganda   \n",
       "3                    Namibia   \n",
       "4                    Nigeria   \n",
       "5                   Colombia   \n",
       "6                      Chile   \n",
       "7                       Fiji   \n",
       "8            Solomon Islands   \n",
       "9      Palestinian Territory   \n",
       "10                   Somalia   \n",
       "11                  Cambodia   \n",
       "12                  Slovakia   \n",
       "13                  Slovenia   \n",
       "14               Afghanistan   \n",
       "15                      Iraq   \n",
       "16                     Nepal   \n",
       "17                 Sri Lanka   \n",
       "18                      Laos   \n",
       "19                   Albania   \n",
       "20                Costa Rica   \n",
       "21            Czech Republic   \n",
       "22            Czech Republic   \n",
       "23                    Canada   \n",
       "24                   Tunisia   \n",
       "25                 Guatemala   \n",
       "26              Burkina Faso   \n",
       "27                    Angola   \n",
       "28                    Panama   \n",
       "29                     Japan   \n",
       "...                      ...   \n",
       "45764            Philippines   \n",
       "45765                  Spain   \n",
       "45766                 Turkey   \n",
       "45767                 Russia   \n",
       "45768         United Kingdom   \n",
       "45769                  Kenya   \n",
       "45770              Australia   \n",
       "45771                 Israel   \n",
       "45772              Australia   \n",
       "45773                Somalia   \n",
       "45774                  India   \n",
       "45775             Kazakhstan   \n",
       "45776            Philippines   \n",
       "45777                   Fiji   \n",
       "45778                Albania   \n",
       "45779     Bosnia-Herzegovina   \n",
       "45780            Philippines   \n",
       "45781         United Kingdom   \n",
       "45782                 Canada   \n",
       "45783                  India   \n",
       "45784                Germany   \n",
       "45785                  China   \n",
       "45786                  China   \n",
       "45787                   Iran   \n",
       "45788                 Greece   \n",
       "45789          United States   \n",
       "45790                 Gambia   \n",
       "45791          United States   \n",
       "45792           Saudi Arabia   \n",
       "45793          United States   \n",
       "\n",
       "                                            article_name revision_id  \\\n",
       "0                                         Bir I of Kanem   355319463   \n",
       "1                      Template:Zimbabwe-politician-stub   391862046   \n",
       "2                        Template:Uganda-politician-stub   391862070   \n",
       "3                       Template:Namibia-politician-stub   391862409   \n",
       "4                       Template:Nigeria-politician-stub   391862819   \n",
       "5                      Template:Colombia-politician-stub   391863340   \n",
       "6                         Template:Chile-politician-stub   391863361   \n",
       "7                          Template:Fiji-politician-stub   391863617   \n",
       "8                      Template:Solomons-politician-stub   391863809   \n",
       "9      Information Minister of the Palestinian Nation...   393276188   \n",
       "10                      Template:Somalia-politician-stub   393298432   \n",
       "11                                               Yos Por   393822005   \n",
       "12                     Template:Slovakia-politician-stub   394482629   \n",
       "13                     Template:Slovenia-politician-stub   394482891   \n",
       "14                  Template:Afghanistan-politician-stub   394580295   \n",
       "15                         Template:Iraq-politician-stub   394580630   \n",
       "16                        Template:Nepal-politician-stub   394580939   \n",
       "17                     Template:SriLanka-politician-stub   394580993   \n",
       "18                         Template:Laos-politician-stub   394581284   \n",
       "19                      Template:Albania-politician-stub   394581557   \n",
       "20                    Template:CostaRica-politician-stub   394587483   \n",
       "21                                          Julius Gregr   395521877   \n",
       "22                                          Edvard Gregr   395526568   \n",
       "23                                   Robert Douglas Cook   401577829   \n",
       "24                      Template:Tunisia-politician-stub   413885084   \n",
       "25                    Template:Guatemala-politician-stub   433871129   \n",
       "26                  Template:BurkinaFaso-politician-stub   433871165   \n",
       "27                       Template:Angola-politician-stub   435008715   \n",
       "28                       Template:Panama-politician-stub   437454659   \n",
       "29                  Template:Japan-politician-1980s-stub   437735138   \n",
       "...                                                  ...         ...   \n",
       "45764                       President of the Philippines   807475001   \n",
       "45765                             Jaime del Burgo Torres   807475042   \n",
       "45766                                       Tunahan Kuzu   807475476   \n",
       "45767                                        Mark Feygin   807475576   \n",
       "45768                                   Bernard Williams   807476248   \n",
       "45769                               Alice Muthoni Wahome   807476386   \n",
       "45770                                          Don Chipp   807476693   \n",
       "45771                                         Avi Gabbay   807476738   \n",
       "45772                                        Greg Wilton   807477468   \n",
       "45773                                         Siad Barre   807477894   \n",
       "45774                             V.P. Deenadayalu Naidu   807478407   \n",
       "45775                               Nursultan Nazarbayev   807478471   \n",
       "45776                                        Aga Muhlach   807479087   \n",
       "45777                                       Elizabeth II   807479170   \n",
       "45778                                         Haki Toska   807479431   \n",
       "45779                                    Hakija Turajlić   807479443   \n",
       "45780                                     King Gutierrez   807479587   \n",
       "45781                                        Mark Boleat   807479783   \n",
       "45782                                        Amir Khadir   807479923   \n",
       "45783                                 Mangal Das Pakvasa   807479965   \n",
       "45784                                       Hakkı Keskin   807480590   \n",
       "45785                                        Yang Jiechi   807481066   \n",
       "45786                                       Li Jiancheng   807481329   \n",
       "45787                                Mahmoud Ahmadinejad   807481450   \n",
       "45788                                 George I of Greece   807481543   \n",
       "45789                                        Hal Bidlack   807481636   \n",
       "45790                                       Yahya Jammeh   807482007   \n",
       "45791                                   Lucius Fairchild   807483006   \n",
       "45792                               Fahd of Saudi Arabia   807483153   \n",
       "45793                                  Francis Fessenden   807483270   \n",
       "\n",
       "      article_quality  population  \n",
       "0                Stub    13707000  \n",
       "1                Stub    17354000  \n",
       "2                Stub    40141000  \n",
       "3                Stub     2482100  \n",
       "4                Stub   181839400  \n",
       "5                Stub    48218000  \n",
       "6                Stub    18025000  \n",
       "7                Stub      867000  \n",
       "8                Stub      641900  \n",
       "9                Stub     4481195  \n",
       "10               Stub    11123000  \n",
       "11               Stub    15417100  \n",
       "12               Stub     5424051  \n",
       "13               Stub     2064000  \n",
       "14               Stub    32247000  \n",
       "15               Stub    37056000  \n",
       "16               Stub    28039000  \n",
       "17               Stub    20868800  \n",
       "18               Stub     6903049  \n",
       "19               Stub     2892000  \n",
       "20               Stub     4832000  \n",
       "21               Stub    10551227  \n",
       "22               Stub    10551227  \n",
       "23               Stub    35833000  \n",
       "24               Stub    11026000  \n",
       "25               Stub    16183752  \n",
       "26               Stub    18450400  \n",
       "27               Stub    25000000  \n",
       "28               Stub     3980000  \n",
       "29               Stub   126866820  \n",
       "...               ...         ...  \n",
       "45764               B   102965300  \n",
       "45765              FA    46368000  \n",
       "45766           Start    78215000  \n",
       "45767               C   144302000  \n",
       "45768              FA    65092000  \n",
       "45769            Stub    44306000  \n",
       "45770               C    23888000  \n",
       "45771               C     8375384  \n",
       "45772           Start    23888000  \n",
       "45773               B    11123000  \n",
       "45774            Stub  1314097616  \n",
       "45775               B    17544274  \n",
       "45776               B   102965300  \n",
       "45777              FA      867000  \n",
       "45778           Start     2892000  \n",
       "45779           Start     3650114  \n",
       "45780           Start   102965300  \n",
       "45781            Stub    65092000  \n",
       "45782               C    35833000  \n",
       "45783            Stub  1314097616  \n",
       "45784           Start    81132000  \n",
       "45785               C  1371920000  \n",
       "45786               C  1371920000  \n",
       "45787               B    78483446  \n",
       "45788              FA    11520785  \n",
       "45789               C   321234172  \n",
       "45790              GA     2021893  \n",
       "45791               C   321234172  \n",
       "45792              GA    31565109  \n",
       "45793               C   321234172  \n",
       "\n",
       "[45794 rows x 5 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Combine the population data and article prediction data\n",
    "\n",
    "combined_data = []\n",
    "for key, value in country_article.items():\n",
    "    country = value[0]\n",
    "    article_name = value[1]\n",
    "    revision_id = key\n",
    "    if revision_id in prediction_data:\n",
    "        article_quality = prediction_data[revision_id]\n",
    "        if country in population_data:\n",
    "            population = population_data[country]\n",
    "            line = [country, article_name, revision_id, article_quality, population]\n",
    "            combined_data.append(line)\n",
    "\n",
    "df = pandas.DataFrame(combined_data, columns=['country', 'article_name', 'revision_id', 'article_quality', 'population'])\n",
    "\n",
    "# Write combined data to CSV\n",
    "df.to_csv('en_wikipedia_article_quality.csv', sep=',', index=False)\n",
    "\n",
    "# Take a look at the combined dataset\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze the bias in data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will analyze the potential bias in the data we just collected through some quantitative analysis. Basically, we will calculate and consider two ratios:\n",
    "1. Number of politician articles to country population\n",
    "2. Number of 'high qualities' ariticles (articles with predictions in either 'FA' or 'GA') to total number of articles\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate the ratio of number of politician articles to country population"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "articles_to_population = pandas.DataFrame(\n",
    "    df.groupby('country').size() / df.groupby('country')['population'].max() * 100\n",
    "    , columns=['Number of articles to population (in percentage)'])\n",
    "articles_to_population = articles_to_population.sort_values(by='Number of articles to population (in percentage)'\n",
    "                                                                   , ascending=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 10 countries in terms of number of politician articles as a proportion of country population"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Number of articles to population (in percentage)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nauru</th>\n",
       "      <td>0.488029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tuvalu</th>\n",
       "      <td>0.466102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>San Marino</th>\n",
       "      <td>0.248485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Monaco</th>\n",
       "      <td>0.105020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Liechtenstein</th>\n",
       "      <td>0.077189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Marshall Islands</th>\n",
       "      <td>0.067273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iceland</th>\n",
       "      <td>0.062268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tonga</th>\n",
       "      <td>0.060987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Andorra</th>\n",
       "      <td>0.043590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Federated States of Micronesia</th>\n",
       "      <td>0.036893</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                Number of articles to population (in percentage)\n",
       "country                                                                         \n",
       "Nauru                                                                   0.488029\n",
       "Tuvalu                                                                  0.466102\n",
       "San Marino                                                              0.248485\n",
       "Monaco                                                                  0.105020\n",
       "Liechtenstein                                                           0.077189\n",
       "Marshall Islands                                                        0.067273\n",
       "Iceland                                                                 0.062268\n",
       "Tonga                                                                   0.060987\n",
       "Andorra                                                                 0.043590\n",
       "Federated States of Micronesia                                          0.036893"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles_to_population.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bottom 10 countries in terms of number of politician articles as a proportion of country population"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Number of articles to population (in percentage)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Bangladesh</th>\n",
       "      <td>0.000202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congo, Dem. Rep. of</th>\n",
       "      <td>0.000194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thailand</th>\n",
       "      <td>0.000172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Zambia</th>\n",
       "      <td>0.000162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Korea, North</th>\n",
       "      <td>0.000156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ethiopia</th>\n",
       "      <td>0.000107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Uzbekistan</th>\n",
       "      <td>0.000093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Indonesia</th>\n",
       "      <td>0.000084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>China</th>\n",
       "      <td>0.000083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>India</th>\n",
       "      <td>0.000075</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Number of articles to population (in percentage)\n",
       "country                                                              \n",
       "Bangladesh                                                   0.000202\n",
       "Congo, Dem. Rep. of                                          0.000194\n",
       "Thailand                                                     0.000172\n",
       "Zambia                                                       0.000162\n",
       "Korea, North                                                 0.000156\n",
       "Ethiopia                                                     0.000107\n",
       "Uzbekistan                                                   0.000093\n",
       "Indonesia                                                    0.000084\n",
       "China                                                        0.000083\n",
       "India                                                        0.000075"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles_to_population.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate the ratio of number of 'high qualities' ariticles to total number of articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gafa_to_articles = pandas.DataFrame(\n",
    "    df[(df.article_quality == 'FA') | (df.article_quality == 'GA')].groupby('country').size() / df.groupby('country').size() * 100\n",
    "    , columns=['Number of \\'high qualities\\' articles to number of articles (in percentage)'])\n",
    "gafa_to_articles = gafa_to_articles.fillna(0).sort_values(\n",
    "    by='Number of \\'high qualities\\' articles to number of articles (in percentage)'\n",
    "                    , ascending=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 10 countries in terms of GA and FA-quality articles as a proportion of all articles about politicians from that country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Number of 'high qualities' articles to number of articles (in percentage)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Korea, North</th>\n",
       "      <td>23.076923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Saudi Arabia</th>\n",
       "      <td>11.764706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Uzbekistan</th>\n",
       "      <td>10.344828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Central African Republic</th>\n",
       "      <td>10.294118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Romania</th>\n",
       "      <td>9.770115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Guinea-Bissau</th>\n",
       "      <td>9.523810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bhutan</th>\n",
       "      <td>9.090909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Vietnam</th>\n",
       "      <td>8.376963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dominica</th>\n",
       "      <td>8.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mauritania</th>\n",
       "      <td>7.692308</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Number of 'high qualities' articles to number of articles (in percentage)\n",
       "country                                                                                            \n",
       "Korea, North                                                      23.076923                        \n",
       "Saudi Arabia                                                      11.764706                        \n",
       "Uzbekistan                                                        10.344828                        \n",
       "Central African Republic                                          10.294118                        \n",
       "Romania                                                            9.770115                        \n",
       "Guinea-Bissau                                                      9.523810                        \n",
       "Bhutan                                                             9.090909                        \n",
       "Vietnam                                                            8.376963                        \n",
       "Dominica                                                           8.333333                        \n",
       "Mauritania                                                         7.692308                        "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gafa_to_articles.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bottom 10 countries in terms of GA and FA-quality articles as a proportion of all articles about politicians from that country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Number of 'high qualities' articles to number of articles (in percentage)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Bahrain</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Seychelles</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Malta</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bahamas</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cape Verde</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Solomon Islands</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Guadeloupe</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>French Guiana</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Finland</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Moldova</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Number of 'high qualities' articles to number of articles (in percentage)\n",
       "country                                                                                   \n",
       "Bahrain                                                        0.0                        \n",
       "Seychelles                                                     0.0                        \n",
       "Malta                                                          0.0                        \n",
       "Bahamas                                                        0.0                        \n",
       "Cape Verde                                                     0.0                        \n",
       "Solomon Islands                                                0.0                        \n",
       "Guadeloupe                                                     0.0                        \n",
       "French Guiana                                                  0.0                        \n",
       "Finland                                                        0.0                        \n",
       "Moldova                                                        0.0                        "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gafa_to_articles.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis and Write-Up\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What I have learned in this project?\n",
    "  \n",
    "This project provides me an opportunity to practice basic skills about data preprocessing in Python such as loading csv files, constructing dictionaries and splitting dataset into batches, and using pandas dataframe to combine datasets and proceeding some advanced aggregations and calculations. Also, I became more familiar with using WikiMedia API in Python. I learned that to improve the efficiency of using API, I could pass in multiple revision id for each API call. It was a really interesting experience to discover the fact that the API may not return the desired result and raise an error sometimes, I spent a large amount of time on debugging this issue. Actually, several days ago I found there are only two revision ids that do not have any article quality prediction, but today I found there are two more revision ids that do not have predictions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What I found from the calculations?  \n",
    "  \n",
    "For the first ratio we calculated, the ratio of number of politician articles to country, I realized that the top-ranked countries are countries that are relatively small regarding both territory and population size, but there are still many people talking about the politicians of those countries becaues small countries tend to have more political and domestic events and changes. \n",
    "\n",
    "In contrast, the bottom-ranked countries have much less percentages of converage of politician articles on English Wikipedia compared to those top-ranked countries (about 100000 times difference). There are two cases: for countries like India and China, even though the numbers of politician articles in English Wikipedia are not small, the final coverage percentage will still be very very small because the population of those countries (the denominators in the caculations) are extremely large. On the other hand, North Korea, even though the country population is not large relatively, the number of politician articles on English Wikipedia is relatively small due to the fact that there is an Internet blocking between North Korea's Internet and Worldwide Internet, so there are less people who are able to publish or even know about North Korea's politics.\n",
    "\n",
    "For the second ratio we calculated, the ratio of umber of 'high qualities' ariticles to total number of articles, we see North Korea gets the first rank. I think there is some bias existed in this result because articles talking about North Korea's politicians may be not be actually written with very good quality, but because of the fact that the articles about North Korea are too rare, the algorithm somehow considers those articles contain extremely precious value and marks them as feature or good articles.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}