tutorials/2-search-and-ir-eval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This is the Indexing and Search metapy tutorial. First, you should read the following tutorial:\n",
    "- [Search Tutorial](https://meta-toolkit.org/search-tutorial.html). Read *Initially setting up the config file* and *Relevance judgements*.\n",
    "\n",
    "First, let's create an index. We will use the AP News dataset. Your current directory should look like this:\n",
    "- `apnews`: AP News 88 dataset in MeTA format.\n",
    "- `queries.txt`: 100 queries, one per line.\n",
    "- `qrels.txt`: Over 10,000 relevance judgements for the queries.\n",
    "- `stopwords.txt`: A file containing stopwords that will not be indexed.\n",
    "- `apnews-config.toml`: A config file with paths set to all the above files, including index and ranker settings.\n",
    "\n",
    "Here's how we can use metapy to create the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import metapy\n",
    "idx = metapy.index.make_inverted_index('apnews-config.toml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This may take a minute at first, since the index needs to be built. Subsequent calls to `make_inverted_index` with this config file will simply load the index, which will not take any time.\n",
    "\n",
    "Here's how we can interact with the index object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "164465"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx.num_docs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "299769"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx.unique_terms()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "526.3216552734375"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx.avg_doc_length()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "86561496"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx.total_corpus_terms()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "All the disk index and inverted index functions from MeTA are implemented in metapy.\n",
    "\n",
    "Let's create a `Ranker` object so we can search the index;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "ranker = metapy.index.OkapiBM25()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, we need a query. Create a `Document` and set its content to our query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "query = metapy.index.Document()\n",
    "query.content('Airbus Subsidies') # query from AP news"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Search our index using our ranker and query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(49687, 20.686737060546875),\n",
       " (8005, 20.367332458496094),\n",
       " (8645, 20.20011329650879),\n",
       " (13158, 20.071775436401367),\n",
       " (10212, 19.91327667236328)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_docs = ranker.score(idx, query, num_results=5)\n",
    "top_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We are returned a ranked list of *(doc_id, score)* pairs. The scores are from the ranker, which in this case was Okapi BM25. Since our `line.toml` file in the AP News dataset has `store-full-text = true`, we can verify the content of our top documents by inspecting the document metadata field \"content\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. A top West German economic official said Sunday that reduction of government subsidies for Airbus Industrie will be a main topic at a planned September meeting ofthe consortium's member nations in Britain . Erich Riedl , parlimentary state secretary ...\n",
      "\n",
      "2. The United States , angry over European subsidies for the Airbus aircraft - manufacturing consortium , is increasing pressure on Airbus nations to abolish or at least reduce the payments , say diplomatic sources . ` ` The Americans at a minimum want ...\n",
      "\n",
      "3. U.S . and European trade official sare holding a new round of talks in the lengthy dispute over government subsidies to the Airbus aircraft manufacturing consortium . But both sides remain far apart on the long - simmering issue of subsidies to Airbu...\n",
      "\n",
      "4. U.S . Trade Representative Clayton Yeutter tol dthe governments of Britain , France , West Germany and Spain onWednesday they are risking a trade war by their ` ` enormous subsidies ' ' to Airbus passenger planes . The major trade bill now before Con...\n",
      "\n",
      "5. The omnibus trade bill pending in Congres swill include a provision allowing the United States to ` ` get tough ' ' with a European airplane manufacturer which domestic aerospace companies claim receives subsidies and engages in unfair trading practi...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for num, (d_id, _) in enumerate(top_docs):\n",
    "    content = idx.metadata(d_id).get('content')\n",
    "    print(\"{}. {}...\\n\".format(num + 1, content[0:250]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Since we have the queries file and relevance judgements, we can do an IR evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "ev = metapy.index.IREval('apnews-config.toml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will loop over the queries file and add each result to the `IREval` object `ev`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query 1 average precision: 1.0\n",
      "Query 2 average precision: 1.0\n",
      "Query 3 average precision: 0.14694444444444443\n",
      "Query 4 average precision: 0.5308730158730158\n",
      "Query 5 average precision: 0.08833333333333333\n",
      "Query 6 average precision: 0.8154365079365078\n",
      "Query 7 average precision: 0.4084126984126984\n",
      "Query 8 average precision: 1.0\n",
      "Query 9 average precision: 0.042222222222222223\n",
      "Query 10 average precision: 0.0\n",
      "Query 11 average precision: 0.9\n",
      "Query 12 average precision: 0.05\n",
      "Query 13 average precision: 0.3\n",
      "Query 14 average precision: 0.0\n",
      "Query 15 average precision: 0.0\n",
      "Query 16 average precision: 0.0\n",
      "Query 17 average precision: 0.0\n",
      "Query 18 average precision: 0.26666666666666666\n",
      "Query 19 average precision: 0.016666666666666666\n",
      "Query 20 average precision: 0.671111111111111\n",
      "Query 21 average precision: 0.5508730158730158\n",
      "Query 22 average precision: 0.0\n",
      "Query 23 average precision: 0.0\n",
      "Query 24 average precision: 0.0\n",
      "Query 25 average precision: 0.0325\n",
      "Query 26 average precision: 0.2\n",
      "Query 27 average precision: 0.4163095238095238\n",
      "Query 28 average precision: 1.0\n",
      "Query 29 average precision: 0.0\n",
      "Query 30 average precision: 0.0\n",
      "Query 31 average precision: 0.24166666666666664\n",
      "Query 32 average precision: 0.8154365079365078\n",
      "Query 33 average precision: 0.12333333333333334\n",
      "Query 34 average precision: 0.05\n",
      "Query 35 average precision: 0.6708730158730158\n",
      "Query 36 average precision: 0.32666666666666666\n",
      "Query 37 average precision: 0.0\n",
      "Query 38 average precision: 0.0125\n",
      "Query 39 average precision: 0.014285714285714285\n",
      "Query 40 average precision: 0.3796428571428571\n",
      "Query 41 average precision: 0.0\n",
      "Query 42 average precision: 0.0\n",
      "Query 43 average precision: 0.4747619047619048\n",
      "Query 44 average precision: 0.0\n",
      "Query 45 average precision: 0.0\n",
      "Query 46 average precision: 0.0125\n",
      "Query 47 average precision: 0.16333333333333333\n",
      "Query 48 average precision: 0.3083333333333333\n",
      "Query 49 average precision: 0.03888888888888888\n",
      "Query 50 average precision: 0.38936507936507936\n",
      "Query 51 average precision: 0.24285714285714288\n",
      "Query 52 average precision: 0.20714285714285713\n",
      "Query 53 average precision: 0.21341269841269842\n",
      "Query 54 average precision: 0.15833333333333333\n",
      "Query 55 average precision: 0.0\n",
      "Query 56 average precision: 0.2596428571428572\n",
      "Query 57 average precision: 0.78\n",
      "Query 58 average precision: 0.03666666666666667\n",
      "Query 59 average precision: 0.0\n",
      "Query 60 average precision: 0.075\n",
      "Query 61 average precision: 1.0\n",
      "Query 62 average precision: 0.2816666666666666\n",
      "Query 63 average precision: 0.0\n",
      "Query 64 average precision: 0.0\n",
      "Query 65 average precision: 0.05\n",
      "Query 66 average precision: 0.0325\n",
      "Query 67 average precision: 0.22063492063492066\n",
      "Query 68 average precision: 0.43555555555555553\n",
      "Query 69 average precision: 0.4397619047619047\n",
      "Query 70 average precision: 0.06666666666666667\n",
      "Query 71 average precision: 0.0\n",
      "Query 72 average precision: 0.09\n",
      "Query 73 average precision: 0.36944444444444446\n",
      "Query 74 average precision: 0.34444444444444444\n",
      "Query 75 average precision: 0.8521031746031745\n",
      "Query 76 average precision: 0.4514285714285714\n",
      "Query 77 average precision: 0.03333333333333333\n",
      "Query 78 average precision: 0.569047619047619\n",
      "Query 79 average precision: 0.34444444444444444\n",
      "Query 80 average precision: 0.639047619047619\n",
      "Query 81 average precision: 0.0\n",
      "Query 82 average precision: 0.18051587301587302\n",
      "Query 83 average precision: 0.5380952380952382\n",
      "Query 84 average precision: 0.6863095238095237\n",
      "Query 85 average precision: 0.3155555555555556\n",
      "Query 86 average precision: 0.16190476190476188\n",
      "Query 87 average precision: 0.7042063492063492\n",
      "Query 88 average precision: 0.0\n",
      "Query 89 average precision: 0.07222222222222222\n",
      "Query 90 average precision: 0.02\n",
      "Query 91 average precision: 0.13333333333333333\n",
      "Query 92 average precision: 0.25\n",
      "Query 93 average precision: 0.03333333333333333\n",
      "Query 94 average precision: 0.21000000000000002\n",
      "Query 95 average precision: 0.13583333333333333\n",
      "Query 96 average precision: 0.44642857142857145\n",
      "Query 97 average precision: 0.15\n",
      "Query 98 average precision: 0.315\n",
      "Query 99 average precision: 0.27166666666666667\n",
      "Query 100 average precision: 0.5258333333333334\n"
     ]
    }
   ],
   "source": [
    "num_results = 10\n",
    "with open('queries.txt') as query_file:\n",
    "    for query_num, line in enumerate(query_file):\n",
    "        query.content(line.strip())\n",
    "        results = ranker.score(idx, query, num_results)                            \n",
    "        avg_p = ev.avg_p(results, query_num, num_results)\n",
    "        print(\"Query {} average precision: {}\".format(query_num + 1, avg_p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Afterwards, we can get the mean average precision of all the queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.26801309523809536"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ev.map()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Try experimenting with different rankers, ranker parameters, tokenization, and filters. What combination give you the best MAP?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Lastly, it's possible to define your own ranking function in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class SimpleRanker(metapy.index.RankingFunction):                                            \n",
    "    \"\"\"                                                                          \n",
    "    Create a new ranking function in Python that can be used in MeTA.             \n",
    "    \"\"\"                                                                          \n",
    "    def __init__(self, some_param=1.0):                                             \n",
    "        self.param = some_param\n",
    "        # You *must* call the base class constructor here!\n",
    "        super(SimpleRanker, self).__init__()                                        \n",
    "                                                                                 \n",
    "    def score_one(self, sd):\n",
    "        \"\"\"\n",
    "        You need to override this function to return a score for a single term.\n",
    "        For fields available in the score_data sd object,\n",
    "        @see https://meta-toolkit.org/doxygen/structmeta_1_1index_1_1score__data.html\n",
    "        \"\"\"\n",
    "        return (self.param + sd.doc_term_count) / (self.param * sd.doc_unique_terms + sd.doc_size)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}