summarization/neural_headline_generation_metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Headline Generation Metrics & Comparisons\n",
    "Although human judgment is the gold standard for assessing the quality of summarization outputs, there are several metrics and comparisons we can gather automatically.\n",
    "In this notebook, we will compare a neural headline generation model with metrics previously reported in these arxiv.org papers:   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "1. <a href=\"https://arxiv.org/abs/1604.01904\">Neural Headline Generation with Sentence-wise Optimization by Ayana, Shiqi Shen, Yu Zhao, Zhiyuan Liu, Maosong Sun</a><br/>    2. <a href=\"https://arxiv.org/abs/1906.03497\">This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation by Rui Zhang, Joel Tetreault</a> <br/>    3. <a href=\"https://arxiv.org/abs/1707.09769\">Low-Resource Neural Headline Generation by Ottokar Tilk, Tanel Alumäe</a> <br/>    4. <a href=\"https://www.semanticscholar.org/paper/HEADS%3A-Headline-Generation-as-Sequence-Prediction-Colmenares-Litvak/a9614b05461bb306cc47c8cd645b9b67bb1227ba\">HEADS: Headline Generation as Sequence Prediction Using an Abstract Feature-Rich Space by Carlos A. Colmenares, Marina Litvak, +1 author Fabrizio Silvestri</a> <br/>    5. <a href=\"https://www.semanticscholar.org/paper/Neural-Headline-Generation-on-Abstract-Meaning-Takase-Suzuki/4d1f12f1a28afc30aab6f5086b3f2e481cf1f49f\">Neural Headline Generation on Abstract Meaning Representation by Sho Takase, Jun Suzuki, +2 authors Masaaki Nagata</a>     \n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "1. <a href=\"https://arxiv.org/abs/1604.01904\">Neural Headline Generation with Sentence-wise Optimization by Ayana, Shiqi Shen, Yu Zhao, Zhiyuan Liu, Maosong Sun</a><br/>    2. <a href=\"https://arxiv.org/abs/1906.03497\">This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation by Rui Zhang, Joel Tetreault</a> <br/>    3. <a href=\"https://arxiv.org/abs/1707.09769\">Low-Resource Neural Headline Generation by Ottokar Tilk, Tanel Alumäe</a> <br/>    4. <a href=\"https://www.semanticscholar.org/paper/HEADS%3A-Headline-Generation-as-Sequence-Prediction-Colmenares-Litvak/a9614b05461bb306cc47c8cd645b9b67bb1227ba\">HEADS: Headline Generation as Sequence Prediction Using an Abstract Feature-Rich Space by Carlos A. Colmenares, Marina Litvak, +1 author Fabrizio Silvestri</a> <br/>    5. <a href=\"https://www.semanticscholar.org/paper/Neural-Headline-Generation-on-Abstract-Meaning-Takase-Suzuki/4d1f12f1a28afc30aab6f5086b3f2e481cf1f49f\">Neural Headline Generation on Abstract Meaning Representation by Sho Takase, Jun Suzuki, +2 authors Masaaki Nagata</a>     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also examine performance with respect to several baseline heuristics detailed in the first paper, namely:\n",
    "* TfIdf (top 8 keywords)\n",
    "* original title\n",
    "\n",
    "The model under consideration is a fine-tuned BERT model trained to predict the words of a headline representing a given document. The model is fine tuned on 347,983 samples. The implementation and the results generated here are based on several Arxiv papers (preprint in progress)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import hist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics\n",
    "import re\n",
    "\n",
    "import pandas as pd\n",
    "from nltk import word_tokenize, sent_tokenize\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import numpy as np\n",
    "from nltk.stem import PorterStemmer\n",
    "\n",
    "stemmer = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title_original</th>\n",
       "      <th>title_generated</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>U.S. Informs Iraqi Government About Reposition...</td>\n",
       "      <td>Some Forces Being Repositioned Inside Iraq</td>\n",
       "      <td>Secretary of Defense Mark Esper announced on M...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ayatollah Weeps And Iranians Rail Against U.S....</td>\n",
       "      <td>Qassem Soleimani The Commander Of Iran's Elite</td>\n",
       "      <td>Vast crowds thronged the streets of Tehran on ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Investor's Naked Selfies Ignite #MeToo Moment:...</td>\n",
       "      <td>Venture Capital Investors Influence Over Startup</td>\n",
       "      <td>\"People would assume that my male colleague wa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Harvey Weinstein Is Charged With Sexual Assaul...</td>\n",
       "      <td>Harvey Weinstein Charged With Sexual Assault</td>\n",
       "      <td>Harvey Weinstein was charged with  four felony...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Australia Wildfires Have Claimed 25 Lives And ...</td>\n",
       "      <td>Deaths Reported In New South</td>\n",
       "      <td>While conditions improved slightly over the we...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      title_original  \\\n",
       "0  U.S. Informs Iraqi Government About Reposition...   \n",
       "1  Ayatollah Weeps And Iranians Rail Against U.S....   \n",
       "2  Investor's Naked Selfies Ignite #MeToo Moment:...   \n",
       "3  Harvey Weinstein Is Charged With Sexual Assaul...   \n",
       "4  Australia Wildfires Have Claimed 25 Lives And ...   \n",
       "\n",
       "                                    title_generated  \\\n",
       "0        Some Forces Being Repositioned Inside Iraq   \n",
       "1    Qassem Soleimani The Commander Of Iran's Elite   \n",
       "2  Venture Capital Investors Influence Over Startup   \n",
       "3      Harvey Weinstein Charged With Sexual Assault   \n",
       "4                      Deaths Reported In New South   \n",
       "\n",
       "                                                text  \n",
       "0  Secretary of Defense Mark Esper announced on M...  \n",
       "1  Vast crowds thronged the streets of Tehran on ...  \n",
       "2  \"People would assume that my male colleague wa...  \n",
       "3  Harvey Weinstein was charged with  four felony...  \n",
       "4  While conditions improved slightly over the we...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "headlines_df = pd.read_csv('headline_generation.sample.csv', sep='\\t')\n",
    "headlines_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rouge recall:\n",
    "#### number of overlapping words / number of words in reference summary\n",
    "\n",
    "Using unigrams is the simplest baseline, others are available https://en.wikipedia.org/wiki/ROUGE_(metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generated Headlines compared to original titles, Rouge-1 recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Headlines compared to original titles, Rouge-1 recall\n",
      "mean: 0.349, stdev: 0.257\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADUlJREFUeJzt3XuoZedZx/Hvr5m2oo02daZDyMVTYQqOFdNwCBFFUyI1F+hUlJBA7TQGR2oqXoow6h8plkKKtEKgpp2SkInYS7zUDiRay1gJiqmd2BqT1NgxnTQzJpnpxVgJVpM+/rFX6jHNzNnn7Fv2M98PbPZa71p7v88758zvvOfde6+TqkKS1NeLFl2AJGm2DHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6Tmtiy6AICtW7fWysrKosuQpKVy7733frmqtq133gsi6FdWVjh06NCiy5CkpZLkkXHOc+lGkpoz6CWpOYNekpoz6CWpOYNekpoz6CWpOYNekpoz6CWpOYNekpp7QXwyVhuzsvfOhfV95MYrF9a3pM1xRi9JzRn0ktTc0i/duIwhSafmjF6SmjPoJak5g16SmjPoJak5g16SmjPoJak5g16SmjPoJak5g16SmjPoJak5g16SmjPoJam5dYM+yXlJPpXkwSQPJPmVof0VST6Z5AvD/VlDe5LclORwkvuSXDjrQUiSTm6cGf3TwNuraidwMXB9kp3AXuBgVe0ADg77AJcDO4bbHuDmqVctSRrbukFfVY9V1T8M218HPg+cA+wC9g+n7QfeOGzvAm6vkXuAlyc5e+qVS5LGsqE1+iQrwGuBTwPbq+qx4dDjwPZh+xzg0TUPOzq0Pfe59iQ5lOTQiRMnNli2JGlcYwd9kpcBfwL8alX9x9pjVVVAbaTjqtpXVatVtbpt27aNPFSStAFjBX2SFzMK+T+sqj8dmp94dklmuD8+tB8Dzlvz8HOHNknSAozzrpsAtwCfr6r3rjl0ANg9bO8GPr6m/c3Du28uBp5cs8QjSZqzcf5m7I8CPwf8U5LPDW2/BdwI3JHkOuAR4Krh2F3AFcBh4Cng2qlWLEnakHWDvqr+BshJDl/6POcXcP2EdUmSpsRPxkpScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtScwa9JDVn0EtSc1sWXYD0Qrey986F9HvkxisX0q/6cUYvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnJdAmMCiPhovSRvhjF6Smls36JPcmuR4kvvXtL0jybEknxtuV6w59ptJDid5KMlPzapwSdJ4xpnR3wZc9jztv1dVFwy3uwCS7ASuBn5weMzvJzljWsVKkjZu3aCvqruBr475fLuAj1TVN6rqi8Bh4KIJ6pMkTWiSNfq3JblvWNo5a2g7B3h0zTlHh7Zvk2RPkkNJDp04cWKCMiRJp7LZd93cDLwTqOH+PcDPb+QJqmofsA9gdXW1NlmH5sw/wiEtn03N6Kvqiap6pqq+CXyQ/1ueOQact+bUc4c2SdKCbCrok5y9ZvengWffkXMAuDrJS5O8CtgB/P1kJUqSJrHu0k2SDwOXAFuTHAVuAC5JcgGjpZsjwC8CVNUDSe4AHgSeBq6vqmdmU7okaRzrBn1VXfM8zbec4vx3Ae+apChJ0vT4yVhJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmNvunBKW5WtSfMJQ6cEYvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLUnEEvSc0Z9JLU3LpBn+TWJMeT3L+m7RVJPpnkC8P9WUN7ktyU5HCS+5JcOMviJUnrG2dGfxtw2XPa9gIHq2oHcHDYB7gc2DHc9gA3T6dMSdJmrRv0VXU38NXnNO8C9g/b+4E3rmm/vUbuAV6e5OxpFStJ2rjNrtFvr6rHhu3Hge3D9jnAo2vOOzq0SZIWZOIXY6uqgNro45LsSXIoyaETJ05MWoYk6SQ2G/RPPLskM9wfH9qPAeetOe/coe3bVNW+qlqtqtVt27ZtsgxJ0no2G/QHgN3D9m7g42va3zy8++Zi4Mk1SzySpAXYst4JST4MXAJsTXIUuAG4EbgjyXXAI8BVw+l3AVcAh4GngGtnULMkaQPWDfqquuYkhy59nnMLuH7SoiRJ0+MnYyWpOYNekpoz6CWpOYNekpoz6CWpOYNekpoz6CWpOYNekpoz6CWpOYNekpoz6CWpOYNekpoz6CWpOYNekppb9zLFkhZjZe+dC+n3yI1XLqRfzY4zeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqzqCXpOYMeklqbsskD05yBPg68AzwdFWtJnkF8FFgBTgCXFVVX5usTEnSZk1jRv+6qrqgqlaH/b3AwaraARwc9iVJCzKLpZtdwP5hez/wxhn0IUka00RLN0ABf5mkgA9U1T5ge1U9Nhx/HNg+YR+SThMre+9cSL9HbrxyIf3Oy6RB/2NVdSzJK4FPJvnntQerqoYfAt8myR5gD8D5558/YRmSpJOZaOmmqo4N98eBjwEXAU8kORtguD9+ksfuq6rVqlrdtm3bJGVIkk5h00Gf5LuSnPnsNvB64H7gALB7OG038PFJi5Qkbd4kSzfbgY8lefZ5PlRVf5HkM8AdSa4DHgGumrxMSdJmbTroq+ph4Iefp/0rwKWTFCVJmh4/GStJzRn0ktScQS9JzRn0ktScQS9JzRn0ktTcpJdAkKSlt6hr7MB8rrPjjF6SmjPoJak5l24k/T+LXMbQbDijl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJas6gl6TmDHpJam5mQZ/ksiQPJTmcZO+s+pEkndpMgj7JGcD7gMuBncA1SXbOoi9J0qnNakZ/EXC4qh6uqv8GPgLsmlFfkqRTmFXQnwM8umb/6NAmSZqzLYvqOMkeYM+w+59JHtrkU20FvjydqpaGYz49OObTQN490Zi/b5yTZhX0x4Dz1uyfO7R9S1XtA/ZN2lGSQ1W1OunzLBPHfHpwzKeHeYx5Vks3nwF2JHlVkpcAVwMHZtSXJOkUZjKjr6qnk7wN+ARwBnBrVT0wi74kSac2szX6qroLuGtWz7/GxMs/S8gxnx4c8+lh5mNOVc26D0nSAnkJBElqbmmCfr1LKiR5aZKPDsc/nWRl/lVO1xhj/vUkDya5L8nBJGO91eqFbNxLZyT5mSSVZOnfoTHOmJNcNXytH0jyoXnXOG1jfG+fn+RTST47fH9fsYg6pyXJrUmOJ7n/JMeT5Kbh3+O+JBdOtYCqesHfGL2g+6/A9wMvAf4R2Pmcc34JeP+wfTXw0UXXPYcxvw74zmH7rafDmIfzzgTuBu4BVhdd9xy+zjuAzwJnDfuvXHTdcxjzPuCtw/ZO4Mii655wzD8OXAjcf5LjVwB/DgS4GPj0NPtflhn9OJdU2AXsH7b/GLg0SeZY47StO+aq+lRVPTXs3sPo8wrLbNxLZ7wTeDfwX/MsbkbGGfMvAO+rqq8BVNXxOdc4beOMuYDvHra/B/i3OdY3dVV1N/DVU5yyC7i9Ru4BXp7k7Gn1vyxBP84lFb51TlU9DTwJfO9cqpuNjV5G4jpGM4Jltu6Yh19pz6uqO+dZ2AyN83V+NfDqJH+b5J4kl82tutkYZ8zvAN6U5Cijd+/98nxKW5iZXjZmYZdA0PQkeROwCvzEomuZpSQvAt4LvGXBpczbFkbLN5cw+q3t7iQ/VFX/vtCqZusa4Laqek+SHwH+IMlrquqbiy5sGS3LjH7dSyqsPSfJFka/7n1lLtXNxjhjJslPAr8NvKGqvjGn2mZlvTGfCbwG+OskRxitZR5Y8hdkx/k6HwUOVNX/VNUXgX9hFPzLapwxXwfcAVBVfwd8B6Pr4HQ11v/3zVqWoB/nkgoHgN3D9s8Cf1XDqxxLat0xJ3kt8AFGIb/s67awzpir6smq2lpVK1W1wuh1iTdU1aHFlDsV43xv/xmj2TxJtjJaynl4nkVO2Thj/hJwKUCSH2AU9CfmWuV8HQDePLz75mLgyap6bFpPvhRLN3WSSyok+R3gUFUdAG5h9OvdYUYvely9uIonN+aYfxd4GfBHw+vOX6qqNyys6AmNOeZWxhzzJ4DXJ3kQeAb4japa2t9Wxxzz24EPJvk1Ri/MvmWZJ25JPszoh/XW4XWHG4AXA1TV+xm9DnEFcBh4Crh2qv0v8b+dJGkMy7J0I0naJINekpoz6CWpOYNekpoz6CWpOYNekpoz6CWpOYNekpr7X6BSNZVKJA8dAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "vals =[]\n",
    "for idx, row in headlines_df.iterrows():\n",
    "    title_original_words = word_tokenize(row['title_original'].lower())\n",
    "    title_generated_words = word_tokenize(row['title_generated'].lower())\n",
    "    title_original_words = [stemmer.stem(word) for word in title_original_words]\n",
    "    title_generated_words = [stemmer.stem(word) for word in title_generated_words]\n",
    "    vals.append(len(set(title_generated_words) & set(title_original_words)) / len(title_generated_words))\n",
    "hist(vals, bins=10)\n",
    "print(f\"Generated Headlines compared to original titles, Rouge-1 recall\\nmean: {statistics.mean(vals):.3f}, stdev: {statistics.stdev(vals):.3f}\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rouge recall figures from papers:\n",
    "* 28.80 - from: Neural Headline Generation on Abstract Meaning Representation\n",
    "* 31.3 (TfIdf keyword score) - from: HEADS: Headline Generation as Sequence Prediction Using an Abstract Feature-Rich Space\n",
    "* 21.34 - from: Low-Resource Neural Headline Generation\n",
    "* 44.0 (F1 Score Email domain) - from: This Email Could Save Your Life\n",
    "* 28.81 - from: Neural Headline Generation with Sentence-wise Optimization\n",
    "\n",
    "Our model appears to be making a strong showing, but the papers also suggest that verticle domain specialization may yield stronger results, as it appears to in the email domain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tf-Idf\n",
    "High values in Tf-Idf correspond well to keywords, and keywords are likely to appear in a title; however a list of keywords rarely makes a suitable title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Tf-Idf feature names: 24,862\n"
     ]
    }
   ],
   "source": [
    "def get_top_tf_idf_words(response, top_n=8):\n",
    "    sorted_nzs = np.argsort(response.data)[:-(top_n+1):-1]\n",
    "    return feature_names[response.indices[sorted_nzs]]\n",
    "\n",
    "# To build our TfIdf calculations, we need to process the entire corpus\n",
    "corpus = [row['text'] for idx, row in headlines_df.iterrows()] \n",
    "tfidf = TfidfVectorizer(stop_words='english')\n",
    "X = tfidf.fit_transform(corpus)\n",
    "feature_names = np.array(tfidf.get_feature_names())\n",
    "print(f\"Number of Tf-Idf feature names: {len(feature_names):,}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An example of a document's top 8 keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['soleimani', 'iran', 'tehran', 'khamenei', 'ghaani', 'wept', 'funeral', 'war']\n",
      "First 200 characters of the doc:\n",
      "Vast crowds thronged the streets of Tehran on Monday to pay respects to Maj. Gen. Qassem Soleimani, the commander of Iran's elite Quds Force who was killed last week in a U.S. drone strike in Baghdad.\n"
     ]
    }
   ],
   "source": [
    "new_doc = headlines_df.iloc[1]['text']\n",
    "responses = tfidf.transform([new_doc])\n",
    "print([get_top_tf_idf_words(response, 8) for response in responses][0].tolist())\n",
    "print(f\"First 200 characters of the doc:\\n{new_doc[:200]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we calculate the top 8 keywords for each document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "top8_keywords = []\n",
    "for idx, row in headlines_df.iterrows():\n",
    "    responses = tfidf.transform([row['text']])\n",
    "    top8_keywords.append([get_top_tf_idf_words(response, 8) for response in responses][0].tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rouge-1 recall of TfIdf keywords and the generated titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rouge-1 recall of TfIdf keywords and the generated titles: \n",
      "mean: 0.282, stdev: 0.195\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAD8FJREFUeJzt3X+MZWV9x/H3RxaxrVTQHcl2WTrUrmnRxoVMCMamRamKS+JiasmSqKshXWOx0dY0We0f2h8kmFZJTCx2DcTVqIC/yqbQWrrSEJuCDoorLKWOuMhuV3YURA2Rdtdv/7gHHenu3jtz753LPPt+JTf3nOc8557vMzP7mTPPPfdsqgpJUrueNukCJEnjZdBLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGrdq0gUArF69uqanpyddhiStKHfeeed3q2qqX7+nRNBPT08zOzs76TIkaUVJ8sAg/Zy6kaTGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxj0lPhmrxZnedtPEjr33yosmdmxJS+MZvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TG9Q36JM9I8qUkX0tyT5K/6NrPTHJHkrkk1yd5etd+Urc+122fHu8QJEnHMsgZ/ePAy6rqRcAG4MIk5wHvBa6qql8HHgEu6/pfBjzStV/V9ZMkTUjfoK+eH3WrJ3aPAl4GfLpr3wFc3C1v6tbptl+QJCOrWJK0KAPN0Sc5IcldwEHgFuCbwPer6lDXZR+wtlteCzwI0G1/FHjOKIuWJA1uoKCvqsNVtQE4HTgX+I1hD5xka5LZJLPz8/PDvpwk6SgWddVNVX0fuBV4MXBKkifuZ386sL9b3g+sA+i2Pwv43hFea3tVzVTVzNTU1BLLlyT1M8hVN1NJTumWfwF4OXAvvcB/bddtC3Bjt7yzW6fb/oWqqlEWLUka3CD/w9QaYEeSE+j9Yrihqv4xyR7guiR/DXwVuKbrfw3wsSRzwMPA5jHULUkaUN+gr6rdwNlHaL+f3nz9k9t/DPzBSKqTJA3NT8ZKUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1Li+QZ9kXZJbk+xJck+St3Xt70myP8ld3WPjgn3emWQuyX1JXjnOAUiSjm3VAH0OAe+oqq8kORm4M8kt3barqupvF3ZOchawGXgB8CvAvyZ5flUdHmXhkqTB9D2jr6oDVfWVbvmHwL3A2mPssgm4rqoer6pvAXPAuaMoVpK0eIuao08yDZwN3NE1vTXJ7iTXJjm1a1sLPLhgt30c+xeDJGmMBg76JM8EPgO8vap+AFwNPA/YABwA3reYAyfZmmQ2yez8/PxidpUkLcJAQZ/kRHoh//Gq+ixAVT1UVYer6ifAh/nZ9Mx+YN2C3U/v2n5OVW2vqpmqmpmamhpmDJKkYxjkqpsA1wD3VtX7F7SvWdDtNcDd3fJOYHOSk5KcCawHvjS6kiVJizHIVTcvAV4PfD3JXV3bu4BLk2wACtgLvBmgqu5JcgOwh94VO5d7xY0kTU7foK+qLwI5wqabj7HPFcAVQ9QlSRoRPxkrSY0bZOpGRzG97aZJlyBJfXlGL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4vkGfZF2SW5PsSXJPkrd17c9OckuSb3TPp3btSfKBJHNJdic5Z9yDkCQd3SBn9IeAd1TVWcB5wOVJzgK2Abuqaj2wq1sHeBWwvntsBa4eedWSpIH1DfqqOlBVX+mWfwjcC6wFNgE7um47gIu75U3AR6vnduCUJGtGXrkkaSCLmqNPMg2cDdwBnFZVB7pN3wFO65bXAg8u2G1f1yZJmoCBgz7JM4HPAG+vqh8s3FZVBdRiDpxka5LZJLPz8/OL2VWStAgDBX2SE+mF/Mer6rNd80NPTMl0zwe79v3AugW7n961/Zyq2l5VM1U1MzU1tdT6JUl9DHLVTYBrgHur6v0LNu0EtnTLW4AbF7S/obv65jzg0QVTPJKkZbZqgD4vAV4PfD3JXV3bu4ArgRuSXAY8AFzSbbsZ2AjMAY8BbxppxZKkRekb9FX1RSBH2XzBEfoXcPmQdUmSRsRPxkpS4wx6SWqcQS9JjTPoJalxBr0kNW6Qyyuf0qa33TTpEo4rk/p6773yookcV2qBZ/SS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUuL5Bn+TaJAeT3L2g7T1J9ie5q3tsXLDtnUnmktyX5JXjKlySNJhBzug/Alx4hParqmpD97gZIMlZwGbgBd0+f5fkhFEVK0lavL5BX1W3AQ8P+HqbgOuq6vGq+hYwB5w7RH2SpCENM0f/1iS7u6mdU7u2tcCDC/rs69okSROy1KC/GngesAE4ALxvsS+QZGuS2SSz8/PzSyxDktTPkoK+qh6qqsNV9RPgw/xsemY/sG5B19O7tiO9xvaqmqmqmampqaWUIUkawJKCPsmaBauvAZ64ImcnsDnJSUnOBNYDXxquREnSMFb165Dkk8D5wOok+4B3A+cn2QAUsBd4M0BV3ZPkBmAPcAi4vKoOj6d0SdIg+gZ9VV16hOZrjtH/CuCKYYqSJI2On4yVpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY3re5ti6Xg3ve2miRx375UXTeS4ao9BrxVhUmErtcCpG0lqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGtc36JNcm+RgkrsXtD07yS1JvtE9n9q1J8kHkswl2Z3knHEWL0nqb5Az+o8AFz6pbRuwq6rWA7u6dYBXAeu7x1bg6tGUKUlaqr5BX1W3AQ8/qXkTsKNb3gFcvKD9o9VzO3BKkjWjKlaStHhLnaM/raoOdMvfAU7rltcCDy7ot69rkyRNyNBvxlZVAbXY/ZJsTTKbZHZ+fn7YMiRJR7HUoH/oiSmZ7vlg174fWLeg3+ld2/9TVduraqaqZqamppZYhiSpn6UG/U5gS7e8BbhxQfsbuqtvzgMeXTDFI0magL63KU7ySeB8YHWSfcC7gSuBG5JcBjwAXNJ1vxnYCMwBjwFvGkPNkqRF6Bv0VXXpUTZdcIS+BVw+bFGSpNHxk7GS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNW7VMDsn2Qv8EDgMHKqqmSTPBq4HpoG9wCVV9chwZUqSlmoUZ/QvraoNVTXTrW8DdlXVemBXty5JmpBxTN1sAnZ0yzuAi8dwDEnSgIYN+gL+JcmdSbZ2badV1YFu+TvAaUMeQ5I0hKHm6IHfrqr9SZ4L3JLkPxdurKpKUkfasfvFsBXgjDPOGLIMSdLRDHVGX1X7u+eDwOeAc4GHkqwB6J4PHmXf7VU1U1UzU1NTw5QhSTqGJQd9kl9KcvITy8ArgLuBncCWrtsW4MZhi5QkLd0wUzenAZ9L8sTrfKKq/jnJl4EbklwGPABcMnyZkqSlWnLQV9X9wIuO0P494IJhipIkjY6fjJWkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1btibmkkak+ltN03kuHuvvGgix9X4eEYvSY0z6CWpcQa9JDXOoJekxhn0ktQ4r7qR9HMmdbUPeMXPuHhGL0mNM+glqXFO3Ug67rU+XeUZvSQ1zqCXpMYZ9JLUOINekhpn0EtS48YW9EkuTHJfkrkk28Z1HEnSsY3l8sokJwAfBF4O7AO+nGRnVe0Zx/EktWGSlzm2bFxn9OcCc1V1f1X9D3AdsGlMx5IkHcO4gn4t8OCC9X1dmyRpmU3sk7FJtgJbu9UfJblviS+1GvjuaKpaMRzz8cExHwfy3qHG/KuDdBpX0O8H1i1YP71r+6mq2g5sH/ZASWarambY11lJHPPxwTEfH5ZjzOOauvkysD7JmUmeDmwGdo7pWJKkYxjLGX1VHUryVuDzwAnAtVV1zziOJUk6trHN0VfVzcDN43r9BYae/lmBHPPxwTEfH8Y+5lTVuI8hSZogb4EgSY1bMUHf75YKSU5Kcn23/Y4k08tf5WgNMOY/TbInye4ku5IMdKnVU9mgt85I8vtJKsmKv0JjkDEnuaT7Xt+T5BPLXeOoDfCzfUaSW5N8tfv53jiJOkclybVJDia5+yjbk+QD3ddjd5JzRlpAVT3lH/Te0P0m8GvA04GvAWc9qc8fAR/qljcD10+67mUY80uBX+yW33I8jLnrdzJwG3A7MDPpupfh+7we+Cpwarf+3EnXvQxj3g68pVs+C9g76bqHHPPvAOcAdx9l+0bgn4AA5wF3jPL4K+WMfpBbKmwCdnTLnwYuSJJlrHHU+o65qm6tqse61dvpfV5hJRv01hl/BbwX+PFyFjcmg4z5D4EPVtUjAFV1cJlrHLVBxlzAL3fLzwL+exnrG7mqug14+BhdNgEfrZ7bgVOSrBnV8VdK0A9yS4Wf9qmqQ8CjwHOWpbrxWOxtJC6jd0awkvUdc/cn7bqqauXuV4N8n58PPD/Jvye5PcmFy1bdeAwy5vcAr0uyj97Ve3+8PKVNzFhvG+N/Dt6AJK8DZoDfnXQt45TkacD7gTdOuJTltore9M359P5quy3Jb1XV9yda1XhdCnykqt6X5MXAx5K8sKp+MunCVqKVckbf95YKC/skWUXvz73vLUt14zHImEnye8CfA6+uqseXqbZx6Tfmk4EXAv+WZC+9ucydK/wN2UG+z/uAnVX1v1X1LeC/6AX/SjXImC8DbgCoqv8AnkHvPjitGujf+1KtlKAf5JYKO4Et3fJrgS9U9y7HCtV3zEnOBv6eXsiv9Hlb6DPmqnq0qlZX1XRVTdN7X+LVVTU7mXJHYpCf7X+gdzZPktX0pnLuX84iR2yQMX8buAAgyW/SC/r5Za1yee0E3tBdfXMe8GhVHRjVi6+IqZs6yi0VkvwlMFtVO4Fr6P15N0fvTY/Nk6t4eAOO+W+AZwKf6t53/nZVvXpiRQ9pwDE3ZcAxfx54RZI9wGHgz6pqxf61OuCY3wF8OMmf0Htj9o0r+cQtySfp/bJe3b3v8G7gRICq+hC99yE2AnPAY8CbRnr8Ffy1kyQNYKVM3UiSlsigl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcf8HQektzr5DbykAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "vals =[]\n",
    "for idx, row in headlines_df.iterrows():\n",
    "    tfidf_keywords = top8_keywords[idx] \n",
    "    title_generated_words = word_tokenize(row['title_generated'].lower())\n",
    "    keywords = [stemmer.stem(word) for word in tfidf_keywords]\n",
    "    title_generated_words = [stemmer.stem(word) for word in title_generated_words]\n",
    "    vals.append(len(set(title_generated_words) & set(keywords)) / len(title_generated_words))\n",
    "hist(vals, bins=10)\n",
    "print(f\"Rouge-1 recall of TfIdf keywords and the generated titles: \\nmean: {statistics.mean(vals):.3f}, stdev: {statistics.stdev(vals):.3f}\")     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rouge-1 recall of TfIdf keywords and the original titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rouge-1 recall of TfIdf keywords and the original titles: \n",
      "mean: 0.203, stdev: 0.112\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEBpJREFUeJzt3X+MZXV9xvH3I6vYKhWUkdBl7aBda7DRxU4JjU2D0lbE1MVoyZJU0dCubbHV1D+6ahPtD1Jsq6SmSrMW4tqosEUN20JtETHGJoADrsAuRRdcwm5WdkRErZF28dM/5qxe12HvvXPnzp3x+34lN3PO9/x65jA8c/bcH5OqQpLUlidMOoAkaflZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGrZl0AIATTzyxpqenJx1DklaV22677etVNbWYbVdE+U9PTzM7OzvpGJK0qiS5f7HbettHkhpk+UtSg/qWf5InJ7k1yZeS7Ery5934qUluSbInydVJntSNH9vN7+mWT4/3W5AkDWuQK/9HgZdW1QuBDcA5Sc4E3g1cVlU/DzwMXNStfxHwcDd+WbeeJGkF6Vv+Ne873ewTu0cBLwWu6ca3Aed10xu7ebrlZyfJkiWWJI1soHv+SY5JshM4CNwA3At8s6oOdavsA9Z202uBBwC65Y8Az1hgn5uTzCaZnZubG+27kCQNZaDyr6rHqmoDcApwBvC8UQ9cVVuraqaqZqamFvUyVUnSIg31ap+q+iZwE/ArwPFJDr9P4BRgfze9H1gH0C1/GvDQkqSVJC2JQV7tM5Xk+G76p4DfAO5m/pfAa7rVLgSu7aZ3dPN0yz9T/qFgSVpRBnmH78nAtiTHMP/LYntV/VuS3cBVSf4K+CJwRbf+FcA/J9kDfAPYNIbcmpDpLddN5Lh7L33FRI4r/aTqW/5VdQdw+gLj9zF////I8e8Bv70k6SRJY+E7fCWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQWsmHUDDm95y3aQjSFrl+l75J1mX5KYku5PsSvLmbvxdSfYn2dk9zu3Z5m1J9iS5J8nLxvkNSJKGN8iV/yHgrVV1e5LjgNuS3NAtu6yq/q535SSnAZuA5wM/C3w6yXOr6rGlDC5JWry+V/5VdaCqbu+mvw3cDaw9yiYbgauq6tGq+iqwBzhjKcJKkpbGUE/4JpkGTgdu6YbelOSOJFcmOaEbWws80LPZPhb4ZZFkc5LZJLNzc3NDB5ckLd7A5Z/kqcDHgbdU1beAy4HnABuAA8B7hjlwVW2tqpmqmpmamhpmU0nSiAYq/yRPZL74P1JVnwCoqger6rGq+j7wQX54a2c/sK5n81O6MUnSCjHIq30CXAHcXVXv7Rk/uWe1VwF3ddM7gE1Jjk1yKrAeuHXpIkuSRjXIq31eDLwWuDPJzm7s7cAFSTYABewF3ghQVbuSbAd2M/9KoYt9pY8krSx9y7+qPg9kgUXXH2WbS4BLRsglSRojP95BkhrkxzuMwI9ZkLRaeeUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqUN/yT7IuyU1JdifZleTN3fjTk9yQ5Cvd1xO68SR5X5I9Se5I8qJxfxOSpOEMcuV/CHhrVZ0GnAlcnOQ0YAtwY1WtB27s5gFeDqzvHpuBy5c8tSRpJH3Lv6oOVNXt3fS3gbuBtcBGYFu32jbgvG56I/DhmnczcHySk5c8uSRp0Ya6559kGjgduAU4qaoOdIu+BpzUTa8FHujZbF83duS+NieZTTI7Nzc3ZGxJ0igGLv8kTwU+Drylqr7Vu6yqCqhhDlxVW6tqpqpmpqamhtlUkjSigco/yROZL/6PVNUnuuEHD9/O6b4e7Mb3A+t6Nj+lG5MkrRCDvNonwBXA3VX13p5FO4ALu+kLgWt7xl/XvernTOCRnttDkqQVYM0A67wYeC1wZ5Kd3djbgUuB7UkuAu4Hzu+WXQ+cC+wBvgu8YUkTS5JG1rf8q+rzQB5n8dkLrF/AxSPmkiSNke/wlaQGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDVoz6QDSIKa3XDexY++99BUTO7Y0Ll75S1KDLH9JapDlL0kN6lv+Sa5McjDJXT1j70qyP8nO7nFuz7K3JdmT5J4kLxtXcEnS4g1y5f8h4JwFxi+rqg3d43qAJKcBm4Dnd9t8IMkxSxVWkrQ0+pZ/VX0O+MaA+9sIXFVVj1bVV4E9wBkj5JMkjcEo9/zflOSO7rbQCd3YWuCBnnX2dWOSpBVkseV/OfAcYANwAHjPsDtIsjnJbJLZubm5RcaQJC3Gosq/qh6sqseq6vvAB/nhrZ39wLqeVU/pxhbax9aqmqmqmampqcXEkCQt0qLKP8nJPbOvAg6/EmgHsCnJsUlOBdYDt44WUZK01Pp+vEOSjwFnAScm2Qe8EzgryQaggL3AGwGqaleS7cBu4BBwcVU9Np7okqTF6lv+VXXBAsNXHGX9S4BLRgklSRov3+ErSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KD/Bu+Uh+T+vvB/u1gjZNX/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGtS3/JNcmeRgkrt6xp6e5IYkX+m+ntCNJ8n7kuxJckeSF40zvCRpcQa58v8QcM4RY1uAG6tqPXBjNw/wcmB999gMXL40MSVJS6lv+VfV54BvHDG8EdjWTW8DzusZ/3DNuxk4PsnJSxVWkrQ01ixyu5Oq6kA3/TXgpG56LfBAz3r7urEDjMn0luvGtWtJ+ok18hO+VVVADbtdks1JZpPMzs3NjRpDkjSExZb/g4dv53RfD3bj+4F1Peud0o39mKraWlUzVTUzNTW1yBiSpMVYbPnvAC7spi8Eru0Zf133qp8zgUd6bg9JklaIvvf8k3wMOAs4Mck+4J3ApcD2JBcB9wPnd6tfD5wL7AG+C7xhDJklSSPqW/5VdcHjLDp7gXULuHjUUJKk8fIdvpLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhq0ZpSNk+wFvg08BhyqqpkkTweuBqaBvcD5VfXwaDElSUtpKa78X1JVG6pqppvfAtxYVeuBG7t5SdIKMo7bPhuBbd30NuC8MRxDkjSCUcu/gP9McluSzd3YSVV1oJv+GnDSiMeQJC2xke75A79aVfuTPBO4Icl/9y6sqkpSC23Y/bLYDPCsZz1rxBiSpGGMdOVfVfu7rweBTwJnAA8mORmg+3rwcbbdWlUzVTUzNTU1SgxJ0pAWXf5JnpLkuMPTwG8CdwE7gAu71S4Erh01pCRpaY1y2+ck4JNJDu/no1X1qSRfALYnuQi4Hzh/9JiSpKW06PKvqvuAFy4w/hBw9iihJEnj5Tt8JalBlr8kNcjyl6QGWf6S1CDLX5IaNOo7fCWNyfSW6yZy3L2XvmIix9Xy8spfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBvl5/pJ+xKT+jgD4twSWk1f+ktQgy1+SGmT5S1KDvOcvqXktPs8xtiv/JOckuSfJniRbxnUcSdLwxlL+SY4B3g+8HDgNuCDJaeM4liRpeOO67XMGsKeq7gNIchWwEdg9puNJ+gkwydsvrRnXbZ+1wAM98/u6MUnSCjCxJ3yTbAY2d7PfSXLPInd1IvD1pUm1rMy9fFZjZjD3cppY5rx7pM1/YbEbjqv89wPreuZP6cZ+oKq2AltHPVCS2aqaGXU/y83cy2c1ZgZzL6fVmBnmcy9223Hd9vkCsD7JqUmeBGwCdozpWJKkIY3lyr+qDiV5E/AfwDHAlVW1axzHkiQNb2z3/KvqeuD6ce2/x8i3jibE3MtnNWYGcy+n1ZgZRsidqlrKIJKkVcDP9pGkBq2a8u/3cRFJjk1ydbf8liTTy5/yxw2Q+9eS3J7kUJLXTCLjkQbI/CdJdie5I8mNSX5uEjmPNEDu309yZ5KdST6/Ut51PuhHoSR5dZJKMvFXpQxwrl+fZK471zuT/O4kch5pkHOd5Pzu53tXko8ud8aFDHC+L+s5119O8s2+O62qFf9g/knje4FnA08CvgScdsQ6fwj8Yze9Cbh6leSeBl4AfBh4zSrJ/BLgp7vpP1hF5/pneqZfCXxqNeTu1jsO+BxwMzCz0jMDrwf+YdLndxG51wNfBE7o5p+5GnIfsf4fMf8im6Pud7Vc+f/g4yKq6n+Bwx8X0WsjsK2bvgY4O0mWMeNC+uauqr1VdQfw/UkEXMAgmW+qqu92szcz/z6OSRsk97d6Zp8CrIQnvAb52Qb4S+DdwPeWM9zjGDTzSjNI7t8D3l9VDwNU1cFlzriQYc/3BcDH+u10tZT/IB8X8YN1quoQ8AjwjGVJ9/hW48dcDJv5IuDfx5poMAPlTnJxknuBvwH+eJmyHU3f3EleBKyrqpXywTeD/oy8urs1eE2SdQssX26D5H4u8Nwk/5Xk5iTnLFu6xzfw/5PdLdhTgc/02+lqKX+tQEl+B5gB/nbSWQZVVe+vqucAfwr82aTz9JPkCcB7gbdOOsuQ/hWYrqoXADfww3+Vr3RrmL/1cxbzV9AfTHL8RBMNZxNwTVU91m/F1VL+fT8uonedJGuApwEPLUu6xzdI7pVmoMxJfh14B/DKqnp0mbIdzbDn+irgvLEmGky/3McBvwh8Nsle4Exgx4Sf9B3k41se6vm5+Cfgl5Yp29EM8jOyD9hRVf9XVV8Fvsz8L4NJGuZnexMD3PIBVs0TvmuA+5j/58zhJzyef8Q6F/OjT/huXw25e9b9ECvjCd9BzvXpzD8BtX7SeYfMvb5n+reA2dWQ+4j1P8vkn/Ad5Fyf3DP9KuDm1XCugXOAbd30iczfbnnGSs/drfc8YC/d+7f67nfS/0GGOAHnMv9b+F7gHd3YXzB/5QnwZOBfgD3ArcCzJ515wNy/zPzVxv8w/y+VXasg86eBB4Gd3WPHpDMPmPvvgV1d5puOVrIrKfcR6068/Ac813/dnesvdef6eZPOPGDuMH+bbTdwJ7Bp0pkH/RkB3gVcOug+fYevJDVotdzzlyQtIctfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QG/T9sKh1gk82NKgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "vals =[]\n",
    "for idx, row in headlines_df.iterrows():\n",
    "    tfidf_keywords = top8_keywords[idx] \n",
    "    title_original_words = word_tokenize(row['title_original'].lower())\n",
    "    keywords = [stemmer.stem(word) for word in tfidf_keywords]\n",
    "    title_original_words = [stemmer.stem(word) for word in title_original_words]\n",
    "    vals.append(len(set(title_original_words) & set(keywords)) / len(title_original_words))\n",
    "hist(vals, bins=10)\n",
    "print(f\"Rouge-1 recall of TfIdf keywords and the original titles: \\nmean: {statistics.mean(vals):.3f}, stdev: {statistics.stdev(vals):.3f}\")     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminary conclusions:\n",
    "* the model produces rouge recall of 35% matching the original title unigrams.\n",
    "* the model produces titles which contain more Tf-Idf keywords than the original titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}