notebooks/Analyze_Model_Outputs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze_Model_Outputs.ipynb\n",
    "### Analyze Model Outputs with Text Extensions for Pandas\n",
    "\n",
    "This Jupyter notebook shows how to use the [Text Extensions for Pandas](https://github.com/CODAIT/text-extensions-for-pandas) library to analyze the outputs of a NLP model on a target corpus.\n",
    "\n",
    "We use the [CoNLL-2003](https://www.clips.uantwerpen.be/conll2003/ner/) corpus as our target corpus, and we use the output of the `bender` team in the original CoNLL 2003 competition as our example model output.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment Setup\n",
    "\n",
    "This notebook requires a Python 3.7 or later environment with `numpy` and `pandas`. \n",
    "\n",
    "The notebook also requires the  `text_extensions_for_pandas` library. You can satisfy this dependency in two ways:\n",
    "\n",
    "* Run `pip install text_extensions_for_pandas` before running this notebook. This command adds the library to your Python environment.\n",
    "* Run this notebook out of your local copy of the Text Extensions for Pandas project's [source tree](https://github.com/CODAIT/text-extensions-for-pandas). In this case, the notebook will use the version of Text Extensions for Pandas in your local source tree **if the package is not installed in your Python environment**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# And of course we need the text_extensions_for_pandas library itself.\n",
    "try:\n",
    "    import text_extensions_for_pandas as tp\n",
    "except ModuleNotFoundError as e:\n",
    "    # If we're running from within the project source tree and the parent Python\n",
    "    # environment doesn't have the text_extensions_for_pandas package, use the\n",
    "    # version in the local source tree.\n",
    "    if not os.getcwd().endswith(\"notebooks\"):\n",
    "        raise e\n",
    "    if \"..\" not in sys.path:\n",
    "        sys.path.insert(0, \"..\")\n",
    "    import text_extensions_for_pandas as tp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the Data Set\n",
    "\n",
    "[CoNLL](https://www.conll.org/), the SIGNLL Conference on Computational Natural Language Learning, is an annual academic conference for natural language processing researchers. Each year's conference features a competition involving a challenging NLP task. The task for the 2003 competition involved identifying mentions of [named entities](https://en.wikipedia.org/wiki/Named-entity_recognition) in English and German news articles from the late 1990's. The corpus for this 2003 competition is one of the most widely-used benchmarks for the performance of named entity recognition models. Current [state-of-the-art results](https://paperswithcode.com/sota/named-entity-recognition-ner-on-conll-2003) on this corpus produce an F1 score (harmonic mean of precision and recall) of 0.93. The best F1 score in the original competition was 0.89.\n",
    "\n",
    "For more information about this data set, we recommend reading the conference paper about the competition results, [\"Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition,\"](https://www.aclweb.org/anthology/W03-0419/).\n",
    "\n",
    "**Note that the data set is licensed for research use only. Be sure to adhere to the terms of the license when using this data set!**\n",
    "\n",
    "The developers of the CoNLL-2003 corpus defined a file format for the corpus, based on the file format used in the earlier [Message Understanding Conference](https://en.wikipedia.org/wiki/Message_Understanding_Conference) competition. This format is generally known as \"CoNLL format\" or \"CoNLL-2003 format\".\n",
    "\n",
    "In the following cell, we use the facilities of Text Extensions for Pandas to download a copy of the CoNLL-2003 data set. Then we read the CoNLL-2003-format file containing the `test` fold of the corpus and translate the data into a collection of Pandas [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) objects, one Dataframe per document. Finally, we display the Dataframe for the first document of the `test` fold of the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>pos</th>\n",
       "      <th>phrase_iob</th>\n",
       "      <th>phrase_type</th>\n",
       "      <th>ent_iob</th>\n",
       "      <th>ent_type</th>\n",
       "      <th>sentence</th>\n",
       "      <th>line_num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[0, 10): '-DOCSTART-'</td>\n",
       "      <td>-X-</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[0, 10): '-DOCSTART-'</td>\n",
       "      <td>1871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[11, 17): 'SOCCER'</td>\n",
       "      <td>NN</td>\n",
       "      <td>B</td>\n",
       "      <td>NP</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[17, 18): '-'</td>\n",
       "      <td>:</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[19, 26): 'ENGLISH'</td>\n",
       "      <td>NNP</td>\n",
       "      <td>B</td>\n",
       "      <td>NP</td>\n",
       "      <td>B</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[27, 31): 'F.A.'</td>\n",
       "      <td>NNP</td>\n",
       "      <td>I</td>\n",
       "      <td>NP</td>\n",
       "      <td>I</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[32, 35): 'CUP'</td>\n",
       "      <td>NNP</td>\n",
       "      <td>I</td>\n",
       "      <td>NP</td>\n",
       "      <td>I</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[36, 42): 'SECOND'</td>\n",
       "      <td>NNP</td>\n",
       "      <td>I</td>\n",
       "      <td>NP</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[43, 48): 'ROUND'</td>\n",
       "      <td>NNP</td>\n",
       "      <td>I</td>\n",
       "      <td>NP</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1879</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[49, 55): 'RESULT'</td>\n",
       "      <td>NNP</td>\n",
       "      <td>I</td>\n",
       "      <td>NP</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[55, 56): '.'</td>\n",
       "      <td>.</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[57, 63): 'LONDON'</td>\n",
       "      <td>NNP</td>\n",
       "      <td>B</td>\n",
       "      <td>NP</td>\n",
       "      <td>B</td>\n",
       "      <td>LOC</td>\n",
       "      <td>[57, 74): 'LONDON 1996-12-06'</td>\n",
       "      <td>1883</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     span  pos phrase_iob phrase_type ent_iob ent_type  \\\n",
       "0   [0, 10): '-DOCSTART-'  -X-          O        None       O     None   \n",
       "1      [11, 17): 'SOCCER'   NN          B          NP       O     None   \n",
       "2           [17, 18): '-'    :          O        None       O     None   \n",
       "3     [19, 26): 'ENGLISH'  NNP          B          NP       B     MISC   \n",
       "4        [27, 31): 'F.A.'  NNP          I          NP       I     MISC   \n",
       "5         [32, 35): 'CUP'  NNP          I          NP       I     MISC   \n",
       "6      [36, 42): 'SECOND'  NNP          I          NP       O     None   \n",
       "7       [43, 48): 'ROUND'  NNP          I          NP       O     None   \n",
       "8      [49, 55): 'RESULT'  NNP          I          NP       O     None   \n",
       "9           [55, 56): '.'    .          O        None       O     None   \n",
       "10     [57, 63): 'LONDON'  NNP          B          NP       B      LOC   \n",
       "\n",
       "                                             sentence  line_num  \n",
       "0                               [0, 10): '-DOCSTART-'      1871  \n",
       "1   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1873  \n",
       "2   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1874  \n",
       "3   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1875  \n",
       "4   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1876  \n",
       "5   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1877  \n",
       "6   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1878  \n",
       "7   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1879  \n",
       "8   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1880  \n",
       "9   [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1881  \n",
       "10                      [57, 74): 'LONDON 1996-12-06'      1883  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download and cache the data set.\n",
    "# NOTE: This data set is licensed for research use only. Be sure to adhere\n",
    "#  to the terms of the license when using this data set!\n",
    "data_set_info = tp.io.conll.maybe_download_conll_data(\"outputs\")\n",
    "data_set_info\n",
    "\n",
    "# Read gold standard data for the \"test\" fold of the corpus.\n",
    "corpus_test_fold = tp.io.conll.conll_2003_to_dataframes(\n",
    "    data_set_info[\"test\"], column_names=[\"pos\", \"phrase\", \"ent\"],\n",
    "    iob_columns=[False, True, True])\n",
    "\n",
    "# Pick some documents to use as examples\n",
    "SHORT_DOC_NUM = 6\n",
    "LONG_DOC_NUM = 0\n",
    "# We use document 6 here because it's short.\n",
    "corpus_test_fold[SHORT_DOC_NUM].head(11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the previous cell corresponds to the first 10 lines of one document in the `test` fold of the corpus in its original format. Here's what the original data looks like:\n",
    "```\n",
    "-DOCSTART- -X- -X- O\n",
    "\n",
    "SOCCER NN I-NP O\n",
    "- : O O\n",
    "ENGLISH NNP I-NP I-MISC\n",
    "F.A. NNP I-NP I-MISC\n",
    "CUP NNP I-NP I-MISC\n",
    "SECOND NNP I-NP O\n",
    "ROUND NNP I-NP O\n",
    "RESULT NNP I-NP O\n",
    ". . O O\n",
    "```\n",
    "\n",
    "Each line represents a single token of the file. The first token of each document is a special token `-DOCSTART-`. Each token is labeled with multiple attributes.\n",
    "\n",
    "The function `tp.io.conll.conll_2003_to_dataframes()` returns a list of DataFrames, one DataFrame per document. The DataFrame above contains the following columns:\n",
    "\n",
    " * **`span`:** The span of the token within a reconstruction of the original document text, with begin and end offsets measured in characters. This column is stored using Text Extensions for Pandas' `SpanArray` extension type.\n",
    " * **`pos`:** Part of speech information for the token, drawn from the second field of each line in the original file. Note that CoNLL-2003 format does not specify the names of metadata fields; this column has the name `pos` because we specified that name in the `column_names` argument to `conll_2003_to_dataframes()`.\n",
    " * **`phrase_iob` and `phrase_type`:** Noun/verb phrase information for the token, drawn from the third field of the original file, in Inside-Outside-Beginning-2 (IOB2) format. The names for these columns come from the `column_names` argument we passed to `conll_2003_to_dataframes()`.\n",
    " * **`ent_iob` and `ent_type`:** Information about named entity mentions at this token offset, in IOB2 format. The names for these columns come from the `column_names` argument we passed to `conll_2003_to_dataframes()`.\n",
    " * **`sentence`:** The span of the sentence containing this token in the reconstructed document text.\n",
    " * **`line_num`:** Which line of the original input file contains this token\n",
    "\n",
    "Note that CoNLL-2003 format uses IOB1 tags for the metadata fields that we call \"ent\" and \"phrase\". The function `conll_2003_to_dataframes()` converts these IOB1 tags to the IOB2 format for ease\n",
    "of consumption. In IOB2 format, every entity starts with a \"begin\" tag, so your code can determine whether a token is the first token in an entity without needing to inspect the previous token. See [the Wikipedia entry for IOB tagging](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging))\n",
    "for more information. \n",
    "\n",
    "We don't need the fields `pos`, `phrase_iob`, `phrase_type` for the remainder of this notebook, so let's drop them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>ent_iob</th>\n",
       "      <th>ent_type</th>\n",
       "      <th>sentence</th>\n",
       "      <th>line_num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[0, 10): '-DOCSTART-'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[0, 10): '-DOCSTART-'</td>\n",
       "      <td>1871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[11, 17): 'SOCCER'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[17, 18): '-'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[19, 26): 'ENGLISH'</td>\n",
       "      <td>B</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[27, 31): 'F.A.'</td>\n",
       "      <td>I</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[32, 35): 'CUP'</td>\n",
       "      <td>I</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[36, 42): 'SECOND'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[43, 48): 'ROUND'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1879</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[49, 55): 'RESULT'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "      <td>1880</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    span ent_iob ent_type  \\\n",
       "0  [0, 10): '-DOCSTART-'       O     None   \n",
       "1     [11, 17): 'SOCCER'       O     None   \n",
       "2          [17, 18): '-'       O     None   \n",
       "3    [19, 26): 'ENGLISH'       B     MISC   \n",
       "4       [27, 31): 'F.A.'       I     MISC   \n",
       "5        [32, 35): 'CUP'       I     MISC   \n",
       "6     [36, 42): 'SECOND'       O     None   \n",
       "7      [43, 48): 'ROUND'       O     None   \n",
       "8     [49, 55): 'RESULT'       O     None   \n",
       "\n",
       "                                            sentence  line_num  \n",
       "0                              [0, 10): '-DOCSTART-'      1871  \n",
       "1  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1873  \n",
       "2  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1874  \n",
       "3  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1875  \n",
       "4  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1876  \n",
       "5  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1877  \n",
       "6  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1878  \n",
       "7  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1879  \n",
       "8  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...      1880  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop unneeded metadata columns\n",
    "if \"pos\" in corpus_test_fold[0].columns:\n",
    "    corpus_test_fold = [\n",
    "        df.drop(columns=[\"pos\", \"phrase_iob\", \"phrase_type\"])\n",
    "        for df in corpus_test_fold\n",
    "    ]\n",
    "corpus_test_fold[SHORT_DOC_NUM].head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the Model's Outputs\n",
    "\n",
    "In this example, we will use the outputs of the \"bender\" team in the original competition as our model outputs.\n",
    "A copy of these outputs is available in this repository under `resources/conll_03/ner/results/bender`.\n",
    "These outputs are also in CoNLL-2003 format, but they do not contain any information about the tokens. For example, \n",
    "here's what the first 10 lines of the example document we've been using in the previous cells looks like\n",
    "in the model outputs:\n",
    "```\n",
    "O\n",
    "\n",
    "O\n",
    "O\n",
    "I-MISC\n",
    "I-MISC\n",
    "I-MISC\n",
    "O\n",
    "O\n",
    "O\n",
    "O\n",
    "```\n",
    "\n",
    "Text Extensions for Pandas includes a function `conll_2003_output_to_dataframes()` that will read this format\n",
    "of model output and merge the tags with the full token information in the original corpus, provided that you\n",
    "have read the original corpus in with `conll_2003_to_dataframes()`. The cell that follows uses this function\n",
    "to read the output of the \"bender\" team, using the `corpus_test_fold` list of DataFrames that we constructed\n",
    "a few cells back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>ent_iob</th>\n",
       "      <th>ent_type</th>\n",
       "      <th>sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[0, 10): '-DOCSTART-'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[0, 10): '-DOCSTART-'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[11, 17): 'SOCCER'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[17, 18): '-'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[19, 26): 'ENGLISH'</td>\n",
       "      <td>B</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[27, 31): 'F.A.'</td>\n",
       "      <td>I</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[32, 35): 'CUP'</td>\n",
       "      <td>I</td>\n",
       "      <td>MISC</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[36, 42): 'SECOND'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[43, 48): 'ROUND'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[49, 55): 'RESULT'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[55, 56): '.'</td>\n",
       "      <td>O</td>\n",
       "      <td>None</td>\n",
       "      <td>[11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    span ent_iob ent_type  \\\n",
       "0  [0, 10): '-DOCSTART-'       O     None   \n",
       "1     [11, 17): 'SOCCER'       O     None   \n",
       "2          [17, 18): '-'       O     None   \n",
       "3    [19, 26): 'ENGLISH'       B     MISC   \n",
       "4       [27, 31): 'F.A.'       I     MISC   \n",
       "5        [32, 35): 'CUP'       I     MISC   \n",
       "6     [36, 42): 'SECOND'       O     None   \n",
       "7      [43, 48): 'ROUND'       O     None   \n",
       "8     [49, 55): 'RESULT'       O     None   \n",
       "9          [55, 56): '.'       O     None   \n",
       "\n",
       "                                            sentence  \n",
       "0                              [0, 10): '-DOCSTART-'  \n",
       "1  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  \n",
       "2  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  \n",
       "3  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  \n",
       "4  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  \n",
       "5  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  \n",
       "6  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  \n",
       "7  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  \n",
       "8  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  \n",
       "9  [11, 56): 'SOCCER- ENGLISH F.A. CUP SECOND ROU...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the outputs of the \"bender\" team in the original competition.\n",
    "bender_output = tp.io.conll.conll_2003_output_to_dataframes(\n",
    "    corpus_test_fold, \"../resources/conll_03/ner/results/bender/eng.testb\")\n",
    "bender_output[SHORT_DOC_NUM].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert IOB-Tagged Data to Lists of Entity Mentions\n",
    "\n",
    "The data we've looked at so far has been in [IOB2 format](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)). \n",
    "Each row of our DataFrame represents a token, and each token is tagged with an entity type (`ent_type`) and an IOB tag (`ent_iob`). The first token of each named entity mention is tagged `B`, while subsequent tokens are tagged `I`. Tokens that aren't part of any named entity are tagged `O`.\n",
    "\n",
    "IOB2 format is a convenient way to represent a corpus, but it is a less useful representation for analyzing the result quality of named entity recognition models. Most tokens in a typical NER corpus will be tagged `O`, any measure of error rate in terms of tokens will over-emphasizing the tokens that are part of entities. Token-level error rate implicitly assigns higher weight to named entity mentions that consist of multiple tokens, further unbalancing error metrics. And most crucially, a naive comparison of IOB tags can result in marking an incorrect answer as correct. Consider a case where the correct sequence of labels is `B, B, I` but the model has output `B, I, I`; in this case, last two tokens of model output are both incorrect (the model has assigned them to the same entity as the first token), but a naive token-level comparison will consider the last token to be correct.\n",
    "\n",
    "The CoNLL 2003 competition used the number of errors in extracting *entire* entity mentions to measure the result quality of the entries. We will use the same metric in this notebook. To compute entity-level errors, we convert the IOB-tagged tokens into pairs of `<entity span,  entity type>`. \n",
    "Text Extensions for Pandas includes a function `iob_to_spans()` that will handle this conversion for you.\n",
    "\n",
    "In the next cell, we use `iob_to_spans()` to convert both the corpus and our example model's output to DataFrames of entity span and type information. Then we display the `<entity span,  entity type>` pairs for the \"bender\" team's output on our example document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>ent_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[19, 35): 'ENGLISH F.A. CUP'</td>\n",
       "      <td>MISC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[57, 63): 'LONDON'</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[88, 110): 'English F.A. Challenge'</td>\n",
       "      <td>MISC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[111, 114): 'Cup'</td>\n",
       "      <td>MISC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[145, 153): 'Plymouth'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[156, 162): 'Exeter'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  span ent_type\n",
       "0         [19, 35): 'ENGLISH F.A. CUP'     MISC\n",
       "1                   [57, 63): 'LONDON'      LOC\n",
       "2  [88, 110): 'English F.A. Challenge'     MISC\n",
       "3                    [111, 114): 'Cup'     MISC\n",
       "4               [145, 153): 'Plymouth'      ORG\n",
       "5                 [156, 162): 'Exeter'      ORG"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert from IOB2-tagged tokens to <span, entity type> pairs.\n",
    "# Again, one DataFrame per document.\n",
    "corpus_spans = [tp.io.conll.iob_to_spans(df) for df in corpus_test_fold]\n",
    "bender_spans = [tp.io.conll.iob_to_spans(df) for df in bender_output]\n",
    "bender_spans[SHORT_DOC_NUM].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each DataFrame in the list `bender_spans` contains two columns, `span` and `ent_type`.\n",
    "The `span` column in the DataFrame has the data type (or [\"dtype\"](https://numpy.org/doc/stable/reference/arrays.dtypes.html), as Pandas\n",
    "and Numpy call them) `TokenSpanDtype`. `TokenSpanDtype` is one of the \n",
    "extension types from Text Extensions for Pandas. \n",
    "The string representation of a Pandas Series shows the dtype of the series\n",
    "(\"TokenSpanDtype\" in this case) on its last line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0           [19, 35): 'ENGLISH F.A. CUP'\n",
       "1                     [57, 63): 'LONDON'\n",
       "2    [88, 110): 'English F.A. Challenge'\n",
       "3                      [111, 114): 'Cup'\n",
       "4                 [145, 153): 'Plymouth'\n",
       "5                   [156, 162): 'Exeter'\n",
       "Name: span, dtype: TokenSpanDtype"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bender_spans[SHORT_DOC_NUM][\"span\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns with a dtype of `TokenSpanDtype` are stored internally using the class\n",
    "`TokenSpanArray`, which is also part of Text Extensions for Pandas.\n",
    "`TokenSpanArray` is a subclass of [`ExtensionArray`](https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.ExtensionArray.html), the base class for custom 1-D array types in Pandas.\n",
    "\n",
    "Pandas stores extension arrays inside the associated Pandas `Series` object\n",
    "for the column. To obtain a reference to a the extension array that backs a \n",
    "column, use the [`array` property of `pandas.Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.array.html#pandas.Series.array):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<TokenSpanArray>\n",
      "[       [19, 35): 'ENGLISH F.A. CUP',                  [57, 63): 'LONDON',\n",
      " [88, 110): 'English F.A. Challenge',                   [111, 114): 'Cup',\n",
      "              [145, 153): 'Plymouth',                [156, 162): 'Exeter']\n",
      "Length: 6, dtype: TokenSpanDtype\n"
     ]
    }
   ],
   "source": [
    "print(bender_spans[SHORT_DOC_NUM][\"span\"].array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the previous cell passed the `TokenSpanArray` to `print()`. The\n",
    "`TokenSpanArray` class can also render itself using [Jupyter Notebook callbacks](https://ipython.readthedocs.io/en/stable/config/integrating.html). To\n",
    "see the HTML representation of the `TokenSpanArray`, pass the array object\n",
    "to Jupyter's [`display()`](https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html#IPython.display.display)\n",
    "function; or make that object be the last line of\n",
    "the cell, as in the following example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"spanArray\">\n",
       "    <div id=\"spans\" \n",
       "     style=\"color: var(--jp-layout-color2); border: 1px solid var(--jp-border-color0); float:left; padding:10px;\">\n",
       "        <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>begin</th>\n",
       "      <th>end</th>\n",
       "      <th>begin_token</th>\n",
       "      <th>end_token</th>\n",
       "      <th>covered_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19</td>\n",
       "      <td>35</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>ENGLISH F.A. CUP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57</td>\n",
       "      <td>63</td>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>LONDON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>88</td>\n",
       "      <td>110</td>\n",
       "      <td>15</td>\n",
       "      <td>18</td>\n",
       "      <td>English F.A. Challenge</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>111</td>\n",
       "      <td>114</td>\n",
       "      <td>18</td>\n",
       "      <td>19</td>\n",
       "      <td>Cup</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>145</td>\n",
       "      <td>153</td>\n",
       "      <td>25</td>\n",
       "      <td>26</td>\n",
       "      <td>Plymouth</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>156</td>\n",
       "      <td>162</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>Exeter</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "    </div>\n",
       "    <div id=\"text\"\n",
       "     style=\"float:right; border: 1px solid var(--jp-border-color0); width: 60%;\">\n",
       "\n",
       "                <div style=\"float:center; padding:10px\">\n",
       "                    <p style=\"font-family:var(--jp-code-font-family); font-size:var(--jp-code-font-size)\">\n",
       "                        -DOCSTART-<br>SOCCER- <span style=\"background-color:rgba(255, 215, 0, 0.5)\">ENGLISH F.A. CUP</span> SECOND ROUND RESULT.<br><span style=\"background-color:rgba(255, 215, 0, 0.5)\">LONDON</span> 1996-12-06<br>Result of an <span style=\"background-color:rgba(255, 215, 0, 0.5)\">English F.A. Challenge</span><br><span style=\"background-color:rgba(255, 215, 0, 0.5)\">Cup</span> second round match on Friday:<br><span style=\"background-color:rgba(255, 215, 0, 0.5)\">Plymouth</span> 4 <span style=\"background-color:rgba(255, 215, 0, 0.5)\">Exeter</span> 1\n",
       "                    </p>\n",
       "                </div>\n",
       "\n",
       "    </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<TokenSpanArray>\n",
       "[       [19, 35): 'ENGLISH F.A. CUP',                  [57, 63): 'LONDON',\n",
       " [88, 110): 'English F.A. Challenge',                   [111, 114): 'Cup',\n",
       "              [145, 153): 'Plymouth',                [156, 162): 'Exeter']\n",
       "Length: 6, dtype: TokenSpanDtype"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bender_spans[SHORT_DOC_NUM][\"span\"].array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The text on the right side of the HTML shows the spans in the context of the reconstructed document text.\n",
    "The table on the left shows detailed information about the spans. \n",
    "Internally, instances of `TokenSpanArray` consist of arrays of begin and end *token* offsets, plus a \n",
    "reference to the tokens. So only the `begin_token` and `end_token` values in the above table are actually \n",
    "stored inside the array. The other attributes in the table, `begin`, `end`, and `covered_text`, are computed\n",
    "on demand.\n",
    "\n",
    "You can obtain a reference to the underlying tokens for a `TokenSpanArray` via the properties\n",
    "`document_tokens` and `tokens`. `document_tokens` returns the single underlying set of tokens\n",
    "for arrays of spans that are all from the same document, while `tokens` returns the (potentially\n",
    "different) backing tokens for each span of an array that may cover spans from multiple documents.\n",
    "\n",
    "In this case, each of our DataFrames spans a single document, so we can use the `document_tokens`\n",
    "property to fetch the tokens of the corresponding document. For example, here are the spans of the \n",
    "first 5 tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"spanArray\">\n",
       "    <div id=\"spans\" \n",
       "     style=\"color: var(--jp-layout-color2); border: 1px solid var(--jp-border-color0); float:left; padding:10px;\">\n",
       "        <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>begin</th>\n",
       "      <th>end</th>\n",
       "      <th>covered_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>-DOCSTART-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11</td>\n",
       "      <td>17</td>\n",
       "      <td>SOCCER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>17</td>\n",
       "      <td>18</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19</td>\n",
       "      <td>26</td>\n",
       "      <td>ENGLISH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27</td>\n",
       "      <td>31</td>\n",
       "      <td>F.A.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "    </div>\n",
       "    <div id=\"text\"\n",
       "     style=\"float:right; border: 1px solid var(--jp-border-color0); width: 60%;\">\n",
       "\n",
       "                <div style=\"float:center; padding:10px\">\n",
       "                    <p style=\"font-family:var(--jp-code-font-family); font-size:var(--jp-code-font-size)\">\n",
       "                        <span style=\"background-color:rgba(255, 215, 0, 0.5)\">-DOCSTART-</span><br><span style=\"background-color:rgba(255, 215, 0, 0.5)\">SOCCER-</span> <span style=\"background-color:rgba(255, 215, 0, 0.5)\">ENGLISH</span> <span style=\"background-color:rgba(255, 215, 0, 0.5)\">F.A.</span> CUP SECOND ROUND RESULT.<br>LONDON 1996-12-06<br>Result of an English F.A. Challenge<br>Cup second round match on Friday:<br>Plymouth 4 Exeter 1\n",
       "                    </p>\n",
       "                </div>\n",
       "\n",
       "    </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<SpanArray>\n",
       "[[0, 10): '-DOCSTART-',    [11, 17): 'SOCCER',         [17, 18): '-',\n",
       "   [19, 26): 'ENGLISH',      [27, 31): 'F.A.']\n",
       "Length: 5, dtype: SpanDtype"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bender_spans[SHORT_DOC_NUM][\"span\"].array.document_tokens[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `SpanArray` object we displayed in the previous cell is the same object\n",
    "that backs the \"span\" column in the token information DataFrame `bender_output[SHORT_DOC_NUM]`\n",
    "from a few cells back:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"spanArray\">\n",
       "    <div id=\"spans\" \n",
       "     style=\"color: var(--jp-layout-color2); border: 1px solid var(--jp-border-color0); float:left; padding:10px;\">\n",
       "        <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>begin</th>\n",
       "      <th>end</th>\n",
       "      <th>covered_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>-DOCSTART-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11</td>\n",
       "      <td>17</td>\n",
       "      <td>SOCCER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>17</td>\n",
       "      <td>18</td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19</td>\n",
       "      <td>26</td>\n",
       "      <td>ENGLISH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27</td>\n",
       "      <td>31</td>\n",
       "      <td>F.A.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "    </div>\n",
       "    <div id=\"text\"\n",
       "     style=\"float:right; border: 1px solid var(--jp-border-color0); width: 60%;\">\n",
       "\n",
       "                <div style=\"float:center; padding:10px\">\n",
       "                    <p style=\"font-family:var(--jp-code-font-family); font-size:var(--jp-code-font-size)\">\n",
       "                        <span style=\"background-color:rgba(255, 215, 0, 0.5)\">-DOCSTART-</span><br><span style=\"background-color:rgba(255, 215, 0, 0.5)\">SOCCER-</span> <span style=\"background-color:rgba(255, 215, 0, 0.5)\">ENGLISH</span> <span style=\"background-color:rgba(255, 215, 0, 0.5)\">F.A.</span> CUP SECOND ROUND RESULT.<br>LONDON 1996-12-06<br>Result of an English F.A. Challenge<br>Cup second round match on Friday:<br>Plymouth 4 Exeter 1\n",
       "                    </p>\n",
       "                </div>\n",
       "\n",
       "    </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<SpanArray>\n",
       "[[0, 10): '-DOCSTART-',    [11, 17): 'SOCCER',         [17, 18): '-',\n",
       "   [19, 26): 'ENGLISH',      [27, 31): 'F.A.']\n",
       "Length: 5, dtype: SpanDtype"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bender_output[SHORT_DOC_NUM][\"span\"].array[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare Model Outputs with the Corpus Labels\n",
    "\n",
    "Now that we have converted our corpus and model outputs to DataFrames of `<span, entity type>` pairs, we can use Pandas to compare the model's output with the labels. \n",
    "\n",
    "There are several ways to compare a set of `<span, entity type>` pairs. You may want to require an exact match of two spans, or you may want to consider partial matches. You might want to require exact matches of the entity types, or you may want to give partial credit to a model output that correctly identifies an entity's span but assigns the wrong entity type to that span.\n",
    "\n",
    "Which of these types of comparisons is the \"right\" way to compare depends on your application. In the cells that follow, we'll show how to use Text Extensions for Pandas to do three different types of span comparison. To simplify the code that follows, we'll start by filtering down to just the Person (`PER`) entities in both output sets. That way we can compare just spans, ignoring the `ent_type` column for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>ent_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[40, 45): 'CHINA'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[66, 77): 'Nadim Ladki'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[482, 495): 'Igor Shkvyrin'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[618, 632): 'Oleg Shatskiku'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>[1079, 1092): 'Takuya Takagi'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>[1148, 1168): 'Hiroshige Yanagimoto'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>[1216, 1227): 'Salem Bitar'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>[1360, 1372): 'Hassan Abbas'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>[1489, 1494): 'Bitar'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>[1503, 1517): 'Nader Jokhadar'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>[1702, 1707): 'Bitar'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>[1761, 1769): 'Shu Kamo'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    span ent_type\n",
       "1                      [40, 45): 'CHINA'      PER\n",
       "2                [66, 77): 'Nadim Ladki'      PER\n",
       "12           [482, 495): 'Igor Shkvyrin'      PER\n",
       "14          [618, 632): 'Oleg Shatskiku'      PER\n",
       "21         [1079, 1092): 'Takuya Takagi'      PER\n",
       "22  [1148, 1168): 'Hiroshige Yanagimoto'      PER\n",
       "24           [1216, 1227): 'Salem Bitar'      PER\n",
       "26          [1360, 1372): 'Hassan Abbas'      PER\n",
       "27                 [1489, 1494): 'Bitar'      PER\n",
       "28        [1503, 1517): 'Nader Jokhadar'      PER\n",
       "33                 [1702, 1707): 'Bitar'      PER\n",
       "35              [1761, 1769): 'Shu Kamo'      PER"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's look at just PER annotations\n",
    "corpus_person = [df[df[\"ent_type\"] == \"PER\"] for df in corpus_spans]\n",
    "bender_person = [df[df[\"ent_type\"] == \"PER\"] for df in bender_spans]\n",
    "\n",
    "# Also, switch to a longer example document to make the comparisons that \n",
    "# follow more interesting.\n",
    "corpus_person[LONG_DOC_NUM]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exact-match comparison between sets of spans\n",
    "\n",
    "Our extension types for spans consider two spans to be equal if they have the same target text and the same begin and end offsets.\n",
    "So you can do exact match comparison between two DataFrames of span data using the standard Pandas \n",
    "[`merge()` function](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>ent_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[66, 77): 'Nadim Ladki'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[618, 632): 'Oleg Shatskiku'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[1079, 1092): 'Takuya Takagi'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[1148, 1168): 'Hiroshige Yanagimoto'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[1216, 1227): 'Salem Bitar'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[1360, 1372): 'Hassan Abbas'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[1503, 1517): 'Nader Jokhadar'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[1761, 1769): 'Shu Kamo'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   span ent_type\n",
       "0               [66, 77): 'Nadim Ladki'      PER\n",
       "1          [618, 632): 'Oleg Shatskiku'      PER\n",
       "2         [1079, 1092): 'Takuya Takagi'      PER\n",
       "3  [1148, 1168): 'Hiroshige Yanagimoto'      PER\n",
       "4           [1216, 1227): 'Salem Bitar'      PER\n",
       "5          [1360, 1372): 'Hassan Abbas'      PER\n",
       "6        [1503, 1517): 'Nader Jokhadar'      PER\n",
       "7              [1761, 1769): 'Shu Kamo'      PER"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_person[LONG_DOC_NUM].merge(bender_person[LONG_DOC_NUM])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relaxed notions of span equality\n",
    "\n",
    "Text Extensions for Pandas also includes a function `contain_join()` for finding pairs of spans where one span in the pair either equals or contains the other span. We can use `contain_join()` to compare two DataFrame columns of spans and find all pairs that satisfy this looser notion of span equivalence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>bender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[66, 77): 'Nadim Ladki'</td>\n",
       "      <td>[66, 77): 'Nadim Ladki'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[482, 495): 'Igor Shkvyrin'</td>\n",
       "      <td>[487, 495): 'Shkvyrin'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[618, 632): 'Oleg Shatskiku'</td>\n",
       "      <td>[618, 632): 'Oleg Shatskiku'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[1079, 1092): 'Takuya Takagi'</td>\n",
       "      <td>[1079, 1092): 'Takuya Takagi'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[1148, 1168): 'Hiroshige Yanagimoto'</td>\n",
       "      <td>[1148, 1168): 'Hiroshige Yanagimoto'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[1216, 1227): 'Salem Bitar'</td>\n",
       "      <td>[1216, 1227): 'Salem Bitar'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[1360, 1372): 'Hassan Abbas'</td>\n",
       "      <td>[1360, 1372): 'Hassan Abbas'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[1503, 1517): 'Nader Jokhadar'</td>\n",
       "      <td>[1503, 1517): 'Nader Jokhadar'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[1761, 1769): 'Shu Kamo'</td>\n",
       "      <td>[1761, 1769): 'Shu Kamo'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 corpus                                bender\n",
       "0               [66, 77): 'Nadim Ladki'               [66, 77): 'Nadim Ladki'\n",
       "1           [482, 495): 'Igor Shkvyrin'                [487, 495): 'Shkvyrin'\n",
       "2          [618, 632): 'Oleg Shatskiku'          [618, 632): 'Oleg Shatskiku'\n",
       "3         [1079, 1092): 'Takuya Takagi'         [1079, 1092): 'Takuya Takagi'\n",
       "4  [1148, 1168): 'Hiroshige Yanagimoto'  [1148, 1168): 'Hiroshige Yanagimoto'\n",
       "5           [1216, 1227): 'Salem Bitar'           [1216, 1227): 'Salem Bitar'\n",
       "6          [1360, 1372): 'Hassan Abbas'          [1360, 1372): 'Hassan Abbas'\n",
       "7        [1503, 1517): 'Nader Jokhadar'        [1503, 1517): 'Nader Jokhadar'\n",
       "8              [1761, 1769): 'Shu Kamo'              [1761, 1769): 'Shu Kamo'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ...give credit for partial matches contained entirely within a true match:\n",
    "tp.spanner.contain_join(corpus_person[LONG_DOC_NUM][\"span\"], \n",
    "                        bender_person[LONG_DOC_NUM][\"span\"], \n",
    "                        \"corpus\", \"bender\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compared with the output of the previous cell, we now have 9 matches instead of 8. The \"bender\" team's model identified the `[487, 495): 'Shkvyrin'` as a Person entity, while the corpus includes the longer span `[482, 495): 'Igor Shkvyrin'`.\n",
    "\n",
    "Text Extensions for Pandas also has second span comparison function `overlap_join()` that finds pairs of spans that overlap. You can use this function to look for equivalent spans between two sets of spans, using this even-looser notion of span equivalence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gold</th>\n",
       "      <th>extracted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[66, 77): 'Nadim Ladki'</td>\n",
       "      <td>[66, 77): 'Nadim Ladki'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[482, 495): 'Igor Shkvyrin'</td>\n",
       "      <td>[487, 495): 'Shkvyrin'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[618, 632): 'Oleg Shatskiku'</td>\n",
       "      <td>[618, 632): 'Oleg Shatskiku'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[1079, 1092): 'Takuya Takagi'</td>\n",
       "      <td>[1079, 1092): 'Takuya Takagi'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[1148, 1168): 'Hiroshige Yanagimoto'</td>\n",
       "      <td>[1148, 1168): 'Hiroshige Yanagimoto'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[1216, 1227): 'Salem Bitar'</td>\n",
       "      <td>[1216, 1227): 'Salem Bitar'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[1360, 1372): 'Hassan Abbas'</td>\n",
       "      <td>[1360, 1372): 'Hassan Abbas'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[1503, 1517): 'Nader Jokhadar'</td>\n",
       "      <td>[1503, 1517): 'Nader Jokhadar'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[1761, 1769): 'Shu Kamo'</td>\n",
       "      <td>[1761, 1769): 'Shu Kamo'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   gold                             extracted\n",
       "0               [66, 77): 'Nadim Ladki'               [66, 77): 'Nadim Ladki'\n",
       "1           [482, 495): 'Igor Shkvyrin'                [487, 495): 'Shkvyrin'\n",
       "2          [618, 632): 'Oleg Shatskiku'          [618, 632): 'Oleg Shatskiku'\n",
       "3         [1079, 1092): 'Takuya Takagi'         [1079, 1092): 'Takuya Takagi'\n",
       "4  [1148, 1168): 'Hiroshige Yanagimoto'  [1148, 1168): 'Hiroshige Yanagimoto'\n",
       "5           [1216, 1227): 'Salem Bitar'           [1216, 1227): 'Salem Bitar'\n",
       "6          [1360, 1372): 'Hassan Abbas'          [1360, 1372): 'Hassan Abbas'\n",
       "7        [1503, 1517): 'Nader Jokhadar'        [1503, 1517): 'Nader Jokhadar'\n",
       "8              [1761, 1769): 'Shu Kamo'              [1761, 1769): 'Shu Kamo'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ...give credit for matches that overlap at all with a true match:\n",
    "tp.spanner.overlap_join(corpus_person[LONG_DOC_NUM][\"span\"], \n",
    "                        bender_person[LONG_DOC_NUM][\"span\"],\n",
    "                        \"gold\", \"extracted\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example document, the \"overlap\" type of span equivalence produces the same result as \"containment\" span equivalence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute Collection-Level Model Accuracy\n",
    "\n",
    "Most benchmark results on the CoNLL-2003 dataset use the accuracy metric from the original competition: F1 score (the harmonic mean of precision and recall) over entity mentions, where an entity mention is considered \"correct\" if it corresponds exactly to an entity mention in the corpus labels.  We can compute this metric using Text Extensions for Pandas' extension types.\n",
    "\n",
    "We start by using Pandas' `merge()` function to find the number of matches between pairs of DataFrames of `<span, label>` pairs. The number of `<span, label>` pairs that exactly match between each pair of DataFrames gives us the number of *true positives* in each document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[8, 31, 32, 20, 5]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_true_positives = [len(corpus_person[i].merge(bender_person[i]).index)\n",
    "                      for i in range(len(corpus_person))]\n",
    "num_true_positives[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The remaining inputs we need are the number of entity mentions the model extracted in each document and the number of mentions that the corpus contains in each document. We can obtain these figures directly from the lengths of our per-document DataFrames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([9, 31, 33, 20, 5], [12, 31, 40, 20, 5])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_extracted = [len(df.index) for df in bender_person]\n",
    "num_entities = [len(df.index) for df in corpus_person]\n",
    "num_extracted[0:5], num_entities[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we combine these three lists of counts into a single DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_num</th>\n",
       "      <th>num_true_positives</th>\n",
       "      <th>num_extracted</th>\n",
       "      <th>num_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>33</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>226</th>\n",
       "      <td>226</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227</th>\n",
       "      <td>227</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>228</th>\n",
       "      <td>228</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229</th>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230</th>\n",
       "      <td>230</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>231 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     doc_num  num_true_positives  num_extracted  num_entities\n",
       "0          0                   8              9            12\n",
       "1          1                  31             31            31\n",
       "2          2                  32             33            40\n",
       "3          3                  20             20            20\n",
       "4          4                   5              5             5\n",
       "..       ...                 ...            ...           ...\n",
       "226      226                   2              2             2\n",
       "227      227                   4              4             6\n",
       "228      228                   4              5             4\n",
       "229      229                   0              0             0\n",
       "230      230                   5              5             9\n",
       "\n",
       "[231 rows x 4 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats_by_doc = pd.DataFrame({\n",
    "    \"doc_num\": np.arange(len(corpus_person)),\n",
    "    \"num_true_positives\": num_true_positives,\n",
    "    \"num_extracted\": num_extracted,\n",
    "    \"num_entities\": num_entities\n",
    "})\n",
    "stats_by_doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard CoNLL-2003 accuracy metric is F1 score over the entire \"test\" fold. We can compute this statistic by aggregating the DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of correct answers: 1421\n",
      "Number of entities identified: 1583\n",
      "Actual number of entities: 1617\n",
      "Precision: 0.8977\n",
      "Recall: 0.8788\n",
      "F1: 0.8881\n"
     ]
    }
   ],
   "source": [
    "total_true_positives = stats_by_doc[\"num_true_positives\"].sum()\n",
    "total_entities = stats_by_doc[\"num_entities\"].sum()\n",
    "total_extracted = stats_by_doc[\"num_extracted\"].sum()\n",
    "\n",
    "precision = total_true_positives / total_extracted\n",
    "recall = total_true_positives / total_entities\n",
    "F1 = 2.0 * (precision * recall) / (precision + recall)\n",
    "print(\n",
    "f\"\"\"Number of correct answers: {total_true_positives}\n",
    "Number of entities identified: {total_extracted}\n",
    "Actual number of entities: {total_entities}\n",
    "Precision: {precision:1.4f}\n",
    "Recall: {recall:1.4f}\n",
    "F1: {F1:1.4f}\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above numbers match up with the official results (last line below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eng.testa\n",
      "processed 51578 tokens with 5942 phrases; found: 5846 phrases; correct: 5280.\n",
      "accuracy:  98.07%; precision:  90.32%; recall:  88.86%; FB1:  89.58\n",
      "              LOC: precision:  93.27%; recall:  93.58%; FB1:  93.42\n",
      "             MISC: precision:  88.51%; recall:  81.02%; FB1:  84.60\n",
      "              ORG: precision:  84.67%; recall:  83.59%; FB1:  84.13\n",
      "              PER: precision:  92.26%; recall:  91.91%; FB1:  92.09\n",
      "eng.testb\n",
      "processed 46666 tokens with 5648 phrases; found: 5548 phrases; correct: 4698.\n",
      "accuracy:  96.80%; precision:  84.68%; recall:  83.18%; FB1:  83.92\n",
      "              LOC: precision:  86.44%; recall:  89.81%; FB1:  88.09\n",
      "             MISC: precision:  78.35%; recall:  73.22%; FB1:  75.70\n",
      "              ORG: precision:  80.27%; recall:  76.16%; FB1:  78.16\n",
      "              PER: precision:  89.77%; recall:  87.88%; FB1:  88.81\n"
     ]
    }
   ],
   "source": [
    "!head -14 ../resources/conll_03/ner/results/bender/conlleval.out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute F1 Score for Each Document\n",
    "\n",
    "In addition to the standard corpus-level accuracy statistics, we can also compute precision, recall, and F1 score for each document by adding some additional columns to our DataFrame `stats_by_doc`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_num</th>\n",
       "      <th>num_true_positives</th>\n",
       "      <th>num_extracted</th>\n",
       "      <th>num_entities</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>12</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.761905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>33</td>\n",
       "      <td>40</td>\n",
       "      <td>0.969697</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.876712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>226</th>\n",
       "      <td>226</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227</th>\n",
       "      <td>227</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>228</th>\n",
       "      <td>228</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.888889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229</th>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230</th>\n",
       "      <td>230</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.555556</td>\n",
       "      <td>0.714286</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>231 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     doc_num  num_true_positives  num_extracted  num_entities  precision  \\\n",
       "0          0                   8              9            12   0.888889   \n",
       "1          1                  31             31            31   1.000000   \n",
       "2          2                  32             33            40   0.969697   \n",
       "3          3                  20             20            20   1.000000   \n",
       "4          4                   5              5             5   1.000000   \n",
       "..       ...                 ...            ...           ...        ...   \n",
       "226      226                   2              2             2   1.000000   \n",
       "227      227                   4              4             6   1.000000   \n",
       "228      228                   4              5             4   0.800000   \n",
       "229      229                   0              0             0        NaN   \n",
       "230      230                   5              5             9   1.000000   \n",
       "\n",
       "       recall        F1  \n",
       "0    0.666667  0.761905  \n",
       "1    1.000000  1.000000  \n",
       "2    0.800000  0.876712  \n",
       "3    1.000000  1.000000  \n",
       "4    1.000000  1.000000  \n",
       "..        ...       ...  \n",
       "226  1.000000  1.000000  \n",
       "227  0.666667  0.800000  \n",
       "228  1.000000  0.888889  \n",
       "229       NaN       NaN  \n",
       "230  0.555556  0.714286  \n",
       "\n",
       "[231 rows x 7 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats_by_doc[\"precision\"] = stats_by_doc[\"num_true_positives\"] / stats_by_doc[\"num_extracted\"]\n",
    "stats_by_doc[\"recall\"] = stats_by_doc[\"num_true_positives\"] / stats_by_doc[\"num_entities\"]\n",
    "stats_by_doc[\"F1\"] = 2.0 * (stats_by_doc[\"precision\"] * stats_by_doc[\"recall\"]) / (stats_by_doc[\"precision\"] + stats_by_doc[\"recall\"])\n",
    "stats_by_doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use Per-Document Accuracy to Identify Problems in Model Outputs\n",
    "\n",
    "We can use these per-document statistics to find documents where this model performed poorly. \n",
    "Here, we use the Pandas `sort_values()` function to identify the top ten most problematic documents by F1 score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_num</th>\n",
       "      <th>num_true_positives</th>\n",
       "      <th>num_extracted</th>\n",
       "      <th>num_entities</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>75</td>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>2</td>\n",
       "      <td>0.095238</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.173913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>138</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>161</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>104</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.444444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>185</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>131</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>85</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>43</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>0.875000</td>\n",
       "      <td>0.437500</td>\n",
       "      <td>0.583333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     doc_num  num_true_positives  num_extracted  num_entities  precision  \\\n",
       "75        75                   2             21             2   0.095238   \n",
       "7          7                   1              2             4   0.500000   \n",
       "8          8                   1              1             5   1.000000   \n",
       "138      138                   2              2            10   1.000000   \n",
       "161      161                   1              3             2   0.333333   \n",
       "104      104                   2              3             6   0.666667   \n",
       "185      185                   1              2             2   0.500000   \n",
       "131      131                   1              2             2   0.500000   \n",
       "85        85                   1              1             3   1.000000   \n",
       "43        43                   7              8            16   0.875000   \n",
       "\n",
       "       recall        F1  \n",
       "75   1.000000  0.173913  \n",
       "7    0.250000  0.333333  \n",
       "8    0.200000  0.333333  \n",
       "138  0.200000  0.333333  \n",
       "161  0.500000  0.400000  \n",
       "104  0.333333  0.444444  \n",
       "185  0.500000  0.500000  \n",
       "131  0.500000  0.500000  \n",
       "85   0.333333  0.500000  \n",
       "43   0.437500  0.583333  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats_by_doc.sort_values(\"F1\").head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's going on with document 75?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h3>PER entities in corpus for document 75:</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>ent_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[53, 70): 'Brendan Intindola'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>[2242, 2252): 'Marc Cohen'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             span ent_type\n",
       "2   [53, 70): 'Brendan Intindola'      PER\n",
       "54     [2242, 2252): 'Marc Cohen'      PER"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<p><h3>PER entities in model outputs for document 75:</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>ent_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[53, 70): 'Brendan Intindola'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[177, 185): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[207, 215): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[264, 272): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[455, 463): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[694, 702): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>[828, 836): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>[1348, 1356): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>[1471, 1475): 'Dome'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>[1578, 1586): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>[1750, 1759): 'Homestake'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>[1944, 1952): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>[2080, 2088): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>[2242, 2252): 'Marc Cohen'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>[2355, 2363): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>[2654, 2662): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>[2890, 2898): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>[2962, 2970): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>[3020, 3028): 'Newmonth'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>[3057, 3065): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>[3143, 3151): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             span ent_type\n",
       "2   [53, 70): 'Brendan Intindola'      PER\n",
       "6          [177, 185): 'Santa Fe'      PER\n",
       "8          [207, 215): 'Santa Fe'      PER\n",
       "10         [264, 272): 'Santa Fe'      PER\n",
       "14         [455, 463): 'Santa Fe'      PER\n",
       "16         [694, 702): 'Santa Fe'      PER\n",
       "18         [828, 836): 'Santa Fe'      PER\n",
       "25       [1348, 1356): 'Santa Fe'      PER\n",
       "28           [1471, 1475): 'Dome'      PER\n",
       "30       [1578, 1586): 'Santa Fe'      PER\n",
       "34      [1750, 1759): 'Homestake'      PER\n",
       "40       [1944, 1952): 'Santa Fe'      PER\n",
       "43       [2080, 2088): 'Santa Fe'      PER\n",
       "54     [2242, 2252): 'Marc Cohen'      PER\n",
       "57       [2355, 2363): 'Santa Fe'      PER\n",
       "59       [2654, 2662): 'Santa Fe'      PER\n",
       "61       [2890, 2898): 'Santa Fe'      PER\n",
       "62       [2962, 2970): 'Santa Fe'      PER\n",
       "64       [3020, 3028): 'Newmonth'      PER\n",
       "65       [3057, 3065): 'Santa Fe'      PER\n",
       "66       [3143, 3151): 'Santa Fe'      PER"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<h3>PER entities in corpus for document 75:</h3>\"))\n",
    "display(corpus_person[75])\n",
    "\n",
    "display(HTML(\"<p><h3>PER entities in model outputs for document 75:</h3>\"))\n",
    "display(bender_person[75])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like this model had trouble with \"Santa Fe\". Let's look at all instances of that string in this document. We can use Text Extensions for Pandas' regular expression support to create spans for every mention of \"Santa Fe\" in document 75:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[36, 44): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[177, 185): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[207, 215): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[264, 272): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[455, 463): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[694, 702): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[828, 836): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[980, 988): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[1348, 1356): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[1578, 1586): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[1944, 1952): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>[2080, 2088): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[2355, 2363): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>[2654, 2662): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[2890, 2898): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[2962, 2970): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[3057, 3065): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[3143, 3151): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>[3434, 3442): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       match\n",
       "0       [36, 44): 'Santa Fe'\n",
       "1     [177, 185): 'Santa Fe'\n",
       "2     [207, 215): 'Santa Fe'\n",
       "3     [264, 272): 'Santa Fe'\n",
       "4     [455, 463): 'Santa Fe'\n",
       "5     [694, 702): 'Santa Fe'\n",
       "6     [828, 836): 'Santa Fe'\n",
       "7     [980, 988): 'Santa Fe'\n",
       "8   [1348, 1356): 'Santa Fe'\n",
       "9   [1578, 1586): 'Santa Fe'\n",
       "10  [1944, 1952): 'Santa Fe'\n",
       "11  [2080, 2088): 'Santa Fe'\n",
       "12  [2355, 2363): 'Santa Fe'\n",
       "13  [2654, 2662): 'Santa Fe'\n",
       "14  [2890, 2898): 'Santa Fe'\n",
       "15  [2962, 2970): 'Santa Fe'\n",
       "16  [3057, 3065): 'Santa Fe'\n",
       "17  [3143, 3151): 'Santa Fe'\n",
       "18  [3434, 3442): 'Santa Fe'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import regex\n",
    "doc_75_tokens = corpus_test_fold[75][\"span\"]\n",
    "\n",
    "# Find all matches of \"Santa Fe\" that start and end on a token boundary\n",
    "santa_fe_mentions = tp.spanner.extract_regex_tok(\n",
    "    doc_75_tokens, regex.compile(r'[Ss]anta\\s+[Ff]e'),\n",
    "    min_len=2, max_len=2)\n",
    "santa_fe_mentions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's line up those regex matches with the  model outputs and corpus labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>match</th>\n",
       "      <th>span</th>\n",
       "      <th>ent_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[177, 185): 'Santa Fe'</td>\n",
       "      <td>[177, 185): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[207, 215): 'Santa Fe'</td>\n",
       "      <td>[207, 215): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[264, 272): 'Santa Fe'</td>\n",
       "      <td>[264, 272): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[455, 463): 'Santa Fe'</td>\n",
       "      <td>[455, 463): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[694, 702): 'Santa Fe'</td>\n",
       "      <td>[694, 702): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[828, 836): 'Santa Fe'</td>\n",
       "      <td>[828, 836): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[1348, 1356): 'Santa Fe'</td>\n",
       "      <td>[1348, 1356): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[1578, 1586): 'Santa Fe'</td>\n",
       "      <td>[1578, 1586): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[1944, 1952): 'Santa Fe'</td>\n",
       "      <td>[1944, 1952): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[2080, 2088): 'Santa Fe'</td>\n",
       "      <td>[2080, 2088): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[2355, 2363): 'Santa Fe'</td>\n",
       "      <td>[2355, 2363): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>[2654, 2662): 'Santa Fe'</td>\n",
       "      <td>[2654, 2662): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[2890, 2898): 'Santa Fe'</td>\n",
       "      <td>[2890, 2898): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>[2962, 2970): 'Santa Fe'</td>\n",
       "      <td>[2962, 2970): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[3057, 3065): 'Santa Fe'</td>\n",
       "      <td>[3057, 3065): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[3143, 3151): 'Santa Fe'</td>\n",
       "      <td>[3143, 3151): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       match                      span ent_type\n",
       "0     [177, 185): 'Santa Fe'    [177, 185): 'Santa Fe'      PER\n",
       "1     [207, 215): 'Santa Fe'    [207, 215): 'Santa Fe'      PER\n",
       "2     [264, 272): 'Santa Fe'    [264, 272): 'Santa Fe'      PER\n",
       "3     [455, 463): 'Santa Fe'    [455, 463): 'Santa Fe'      PER\n",
       "4     [694, 702): 'Santa Fe'    [694, 702): 'Santa Fe'      PER\n",
       "5     [828, 836): 'Santa Fe'    [828, 836): 'Santa Fe'      PER\n",
       "6   [1348, 1356): 'Santa Fe'  [1348, 1356): 'Santa Fe'      PER\n",
       "7   [1578, 1586): 'Santa Fe'  [1578, 1586): 'Santa Fe'      PER\n",
       "8   [1944, 1952): 'Santa Fe'  [1944, 1952): 'Santa Fe'      PER\n",
       "9   [2080, 2088): 'Santa Fe'  [2080, 2088): 'Santa Fe'      PER\n",
       "10  [2355, 2363): 'Santa Fe'  [2355, 2363): 'Santa Fe'      PER\n",
       "11  [2654, 2662): 'Santa Fe'  [2654, 2662): 'Santa Fe'      PER\n",
       "12  [2890, 2898): 'Santa Fe'  [2890, 2898): 'Santa Fe'      PER\n",
       "13  [2962, 2970): 'Santa Fe'  [2962, 2970): 'Santa Fe'      PER\n",
       "14  [3057, 3065): 'Santa Fe'  [3057, 3065): 'Santa Fe'      PER\n",
       "15  [3143, 3151): 'Santa Fe'  [3143, 3151): 'Santa Fe'      PER"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "santa_fe_bender = pd.merge(santa_fe_mentions, bender_person[75], \n",
    "                           left_on=\"match\", right_on=\"span\")\n",
    "santa_fe_bender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>match</th>\n",
       "      <th>span</th>\n",
       "      <th>ent_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[36, 44): 'Santa Fe'</td>\n",
       "      <td>[36, 44): 'Santa Fe'</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[207, 215): 'Santa Fe'</td>\n",
       "      <td>[207, 215): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[264, 272): 'Santa Fe'</td>\n",
       "      <td>[264, 272): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[455, 463): 'Santa Fe'</td>\n",
       "      <td>[455, 463): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[694, 702): 'Santa Fe'</td>\n",
       "      <td>[694, 702): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[828, 836): 'Santa Fe'</td>\n",
       "      <td>[828, 836): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[980, 988): 'Santa Fe'</td>\n",
       "      <td>[980, 988): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[1348, 1356): 'Santa Fe'</td>\n",
       "      <td>[1348, 1356): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[1578, 1586): 'Santa Fe'</td>\n",
       "      <td>[1578, 1586): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[1944, 1952): 'Santa Fe'</td>\n",
       "      <td>[1944, 1952): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[2080, 2088): 'Santa Fe'</td>\n",
       "      <td>[2080, 2088): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>[2355, 2363): 'Santa Fe'</td>\n",
       "      <td>[2355, 2363): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[2654, 2662): 'Santa Fe'</td>\n",
       "      <td>[2654, 2662): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>[2890, 2898): 'Santa Fe'</td>\n",
       "      <td>[2890, 2898): 'Santa Fe'</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[2962, 2970): 'Santa Fe'</td>\n",
       "      <td>[2962, 2970): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[3057, 3065): 'Santa Fe'</td>\n",
       "      <td>[3057, 3065): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[3143, 3151): 'Santa Fe'</td>\n",
       "      <td>[3143, 3151): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[3434, 3442): 'Santa Fe'</td>\n",
       "      <td>[3434, 3442): 'Santa Fe'</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       match                      span ent_type\n",
       "0       [36, 44): 'Santa Fe'      [36, 44): 'Santa Fe'      LOC\n",
       "1     [207, 215): 'Santa Fe'    [207, 215): 'Santa Fe'      ORG\n",
       "2     [264, 272): 'Santa Fe'    [264, 272): 'Santa Fe'      ORG\n",
       "3     [455, 463): 'Santa Fe'    [455, 463): 'Santa Fe'      ORG\n",
       "4     [694, 702): 'Santa Fe'    [694, 702): 'Santa Fe'      ORG\n",
       "5     [828, 836): 'Santa Fe'    [828, 836): 'Santa Fe'      ORG\n",
       "6     [980, 988): 'Santa Fe'    [980, 988): 'Santa Fe'      ORG\n",
       "7   [1348, 1356): 'Santa Fe'  [1348, 1356): 'Santa Fe'      ORG\n",
       "8   [1578, 1586): 'Santa Fe'  [1578, 1586): 'Santa Fe'      ORG\n",
       "9   [1944, 1952): 'Santa Fe'  [1944, 1952): 'Santa Fe'      ORG\n",
       "10  [2080, 2088): 'Santa Fe'  [2080, 2088): 'Santa Fe'      ORG\n",
       "11  [2355, 2363): 'Santa Fe'  [2355, 2363): 'Santa Fe'      ORG\n",
       "12  [2654, 2662): 'Santa Fe'  [2654, 2662): 'Santa Fe'      ORG\n",
       "13  [2890, 2898): 'Santa Fe'  [2890, 2898): 'Santa Fe'      LOC\n",
       "14  [2962, 2970): 'Santa Fe'  [2962, 2970): 'Santa Fe'      ORG\n",
       "15  [3057, 3065): 'Santa Fe'  [3057, 3065): 'Santa Fe'      ORG\n",
       "16  [3143, 3151): 'Santa Fe'  [3143, 3151): 'Santa Fe'      ORG\n",
       "17  [3434, 3442): 'Santa Fe'  [3434, 3442): 'Santa Fe'      ORG"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "santa_fe_corpus = pd.merge(santa_fe_mentions, corpus_spans[75], \n",
    "                           left_on=\"match\", right_on=\"span\")\n",
    "santa_fe_corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll compare the above two sets of spans to each other to create a picture of how the corpus and the \"bender\" team's output treated each of our regular expression matches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>span</th>\n",
       "      <th>ent_type_bender</th>\n",
       "      <th>ent_type_corpus</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[177, 185): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[207, 215): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[264, 272): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[455, 463): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[694, 702): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[828, 836): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[1348, 1356): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[1578, 1586): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[1944, 1952): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[2080, 2088): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[2355, 2363): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>[2654, 2662): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[2890, 2898): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>[2962, 2970): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[3057, 3065): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[3143, 3151): 'Santa Fe'</td>\n",
       "      <td>PER</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        span ent_type_bender ent_type_corpus\n",
       "0     [177, 185): 'Santa Fe'             PER             NaN\n",
       "1     [207, 215): 'Santa Fe'             PER             ORG\n",
       "2     [264, 272): 'Santa Fe'             PER             ORG\n",
       "3     [455, 463): 'Santa Fe'             PER             ORG\n",
       "4     [694, 702): 'Santa Fe'             PER             ORG\n",
       "5     [828, 836): 'Santa Fe'             PER             ORG\n",
       "6   [1348, 1356): 'Santa Fe'             PER             ORG\n",
       "7   [1578, 1586): 'Santa Fe'             PER             ORG\n",
       "8   [1944, 1952): 'Santa Fe'             PER             ORG\n",
       "9   [2080, 2088): 'Santa Fe'             PER             ORG\n",
       "10  [2355, 2363): 'Santa Fe'             PER             ORG\n",
       "11  [2654, 2662): 'Santa Fe'             PER             ORG\n",
       "12  [2890, 2898): 'Santa Fe'             PER             LOC\n",
       "13  [2962, 2970): 'Santa Fe'             PER             ORG\n",
       "14  [3057, 3065): 'Santa Fe'             PER             ORG\n",
       "15  [3143, 3151): 'Santa Fe'             PER             ORG"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = [\"span\", \"ent_type\"]\n",
    "matching_spans = pd.merge(santa_fe_bender[cols], santa_fe_corpus[cols],\n",
    "                          on=\"span\", how=\"left\", suffixes=[\"_bender\", \"_corpus\"])\n",
    "matching_spans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of 15 instances of \"Santa Fe\" that the \"bender\" model tagged as `PER`, 14 are tagged in the corpus as `ORG` or `LOC`. What's going on with the 15th? Let's look at the context of that span in the reconstructed document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'... the most likely white knight buyer for [Santa Fe] Pacific Gold Corp if Santa Fe rejects u...'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matching_spans.iloc[0][\"span\"].context()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It like that span is part of a larger entity, \"Santa Fe Pacific Gold Corp\". We can verify this fact by \n",
    "using `text_extensions_for_pandas.overlap_join()` to find the span in the corpus labels that overlaps\n",
    "with the regular expression match:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>regex_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[177, 203): 'Santa Fe Pacific Gold Corp'</td>\n",
       "      <td>[177, 185): 'Santa Fe'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     corpus             regex_match\n",
       "0  [177, 203): 'Santa Fe Pacific Gold Corp'  [177, 185): 'Santa Fe'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tp.spanner.overlap_join(\n",
    "    corpus_spans[75][\"span\"], matching_spans.iloc[[0]][\"span\"],\n",
    "    \"corpus\", \"regex_match\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}