probabilistic_language_modeling/collocation_extractions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collocation Extraction Methodologies Compared\n",
    "\n",
    "In corpus linguistics, a collocation is a series of words or terms that co-occur more often than would be expected by chance. In phraseology, a collocation is a type of compositional phraseme, meaning that it can be understood from the words that make it up. This contrasts with an idiom, where the meaning of the whole cannot be inferred from its parts, and may be completely unrelated. \n",
    "\n",
    "An example of a phraseological collocation is the expression strong tea.[1] While the same meaning could be conveyed by the roughly equivalent powerful tea, this expression is considered excessive and awkward by English speakers. Conversely, a corresponding expression in technology, powerful computer, is preferred over strong computer.\n",
    "\n",
    "* https://en.wikipedia.org/wiki/Collocation\n",
    "\n",
    "#### This notebook examines the different ways a collocation can be calculated (Loglikelihood ratio of bigram probabilities, Pointwise mutual information) and some problems and solutions for extending beyond bigram collocations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from random import sample\n",
    "\n",
    "import stanza\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from nltk import ngrams\n",
    "from nltk.corpus import brown\n",
    "from tqdm import tqdm\n",
    "from scipy.stats import binom, chi2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binomial Probabilty Mass Function used for likelihood matching\n",
    "Often, for computing the probability of a collocation, we assume a binomial distribution and coincedental matching is an independent probability. Here's quick refresher with a concrete example.\n",
    "\n",
    "Suppose a die is tossed 5 times. \n",
    "\n",
    "What is the probability of getting exactly 2 fours?\n",
    "\n",
    "Solution: \n",
    "* This is a binomial experiment in which the number of *trials* is equal to 5,\n",
    "* the number of *successes* is equal to 2,\n",
    "* and the *probability of success* on a single trial is 1/6 or about 0.167.\n",
    "* Therefore, the binomial probability is:         \n",
    "binom.pmf(2, 5, 0.167)\n",
    "\n",
    "0.16120107977393003\n",
    "\n",
    "* https://en.wikipedia.org/wiki/Binomial_distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Log-Likelihood Ratio Test\n",
    "The Log-likelihood ratio is the ratio of two hypothesis on the probability of words occuring together; a formulation of the Null hypothesis and a hypothesis based on observed values. These hypothesis and ratios allow us to formulate the confidence with which we believe a set of words form a collocation.\n",
    "\n",
    "$log \\lambda = log \\frac{L(H_0)}{L(H_1)}$\n",
    "\n",
    "_Hypothesis 0_\n",
    "\n",
    "$P(w^2 | w^1) = p = P(w^2 | \\neg w^1)$ \n",
    "\n",
    "A formulation of independence (the occurrence of $w^2$ is independent of previous occurrence of $w^1$); our Null Hypothesis.\n",
    "\n",
    "_Hypothesis 1_\n",
    "\n",
    "$P(w^2 | w^1) = p_1 \\neq p_2 = P(w^2 | \\neg w^1)$ \n",
    "\n",
    "A formulation of dependence, and good evidence of a collocation.\n",
    "\n",
    "Using maximum Likelihood estimates for $p$, $p_1$, $p_2$, and $c_1$, $c_2$, $c_{12}$ as the number of occurrences for $w^1$, $w^2$, $w^{12}$ in the corpus\n",
    "\n",
    "$P = \\frac{c_2}{N}$\n",
    "$P_1 = \\frac{c_{12}}{c_1}$\n",
    "$P_2 = \\frac{c_2 - c_{12}}{N - c_1}$\n",
    "\n",
    "Using the binomial probabilties these are formulated as:\n",
    "\n",
    "$\\Large log \\frac{b(c_{12}; c_1, p) b(c_2 - c_{12}; N - c_1, p)}{b(c_{12}; c_1, p_1) b(c_2 - c_{12}; N - c_1, p_2)}$\n",
    "\n",
    "in Manning and Schütze (1999) they simplify as:\n",
    "\n",
    "$log L(c_{12}; c_1, p) + log L(c_2 - c_{12}; N - c_1, p) - log L(c_{12}; c_1, p_1) - log L(c_2 - c_{12}; N - c_1, p_2)$\n",
    "\n",
    "The notation may be less confusing if you think in terms:\n",
    "\n",
    "$p = P(w^2)$\n",
    "\n",
    "$p_1 = P(w^2|w^1)$\n",
    "\n",
    "$p_2 = P(w^2|\\neg w^1)$\n",
    "\n",
    "The entire space is the pairs ($p_1, p_2$) for the probability of $w^2$ occurring when $w^1$ preceded $p_1$, and $w^2$ occurring when a different word preceded ($p_2$).\n",
    "\n",
    "see also: https://en.wikipedia.org/wiki/Likelihood-ratio_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Degree of freedom is calculated by using the following formula:\n",
    "* DF = (r-1)(c-1)\n",
    "* DF = Degree of freedom\n",
    "* r = number of rows\n",
    "* c = number of columns\n",
    "\n",
    "In null hypothesis significance testing, the p-value is the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct. A very small p-value means that such an extreme observed outcome would be very unlikely under the null hypothesis.\n",
    "* https://en.wikipedia.org/wiki/P-value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loglikelihood_ratio(c1, c2, c12, N):\n",
    "    \"\"\"\n",
    "    Compute the ratio of two hypotheses of likelihood and return the ratio.\n",
    "    \n",
    "    Under the Independence hypothesis (H0) we assume that there is \n",
    "    no association between w1 and w2, i.e. they are independent: \n",
    "    let P(w1) and P(w2) be probabilities that \n",
    "    a random token in a text is w1 and w2 respectfully and \n",
    "    P(w1,w2) is the probability that (w1,w2) occur together in the text \n",
    "    (i.e. one follows another) so under H0, P(w1,w2) = P(w1)P(w2)\n",
    "    we can compute the observed probability of P(w1,w2) \n",
    "    and compare it with the probability under H0\n",
    "    if these probabilities are significantly different from each other,\n",
    "    then (w1,w2) is a collocation.\n",
    "    \n",
    "    The formula here and test verification values are taken from \n",
    "    Manning & Schūtze _Foundations of Statistical Natural Language Processing_ p.172-175\n",
    "    \n",
    "    Parameters:\n",
    "    c1: count of word 1\n",
    "    c2: count of word 2\n",
    "    c12: count of bigram (w1, w2)\n",
    "    N: the number of words in the corpus\n",
    "    \n",
    "    The value: -2 * loglikelihood_ratio is asymptotically Chi-squared distributed \n",
    "    so we can use Chi-squared table values to test the Null-Hypothesis\n",
    "    against the second Hypothesis--the observed values--as represented by the ratio.\n",
    "    \n",
    "    The following example is taken from Manning and Schütze _Statistical NLP_ p.107\n",
    "    # 1990 NYT data\n",
    "    >>> N = 14_307_668\n",
    "    >>> c1 = 932 # powerful\n",
    "    >>> c2 = 934 # computers\n",
    "    >>> c12 = 10 # bigram count \n",
    "    >>> res = -2 * loglikelihood_ratio(c1, c2, c12, N)\n",
    "    >>> res \n",
    "    82.37586050140558\n",
    "    >>> # e.g. for a alpha value of: 0.005, \n",
    "    >>> # 1 Degree of freedom requires a chi-squared value of 7.88\n",
    "    >>> from scipy.stats import chi2\n",
    "    >>> print ('p-value: %.30f' % chi2.sf(res, 1)) # 1 degrees of freedom\n",
    "    p-value: 0.000000000000000000112519634099\n",
    "    \n",
    "    \"\"\"\n",
    "    p = c2 / N\n",
    "    p1 = c12 / c1\n",
    "    p2 = (c2 - c12) / (N - c1)   \n",
    "    # We proactively trap a runtimeWarning: divide by zero encountered in log,\n",
    "    # which may occur with extreme collocations\n",
    "    import warnings\n",
    "    with warnings.catch_warnings(): # this will reset our filterwarnings setting\n",
    "        warnings.filterwarnings('error')\n",
    "        try:\n",
    "            return (np.log(binom.pmf(c12, c1, p)) \n",
    "                    + np.log(binom.pmf(c2 - c12, N - c1, p)) \n",
    "                    - np.log(binom.pmf(c12, c1, p1) )\n",
    "                    - np.log(binom.pmf(c2 - c12, N - c1, p2)))             \n",
    "        except Warning:\n",
    "            return np.inf "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "82.37586050140521 p-value: 0.000000000000000000112519634099\n"
     ]
    }
   ],
   "source": [
    "# 1990 NYT data\n",
    "N = 14_307_668\n",
    "c1 = 932 # powerful\n",
    "c2 = 934 # computers\n",
    "c12 = 10\n",
    "res = -2 * loglikelihood_ratio(c1, c2, c12, N)\n",
    "# Note: we multiply the ratio by -2 and \n",
    "# the value is directly comparable to Chi-squared values\n",
    "# e.g. for a alpha value of: 0.005, 1 Degree of freedom \n",
    "# requires a chi-squared value of 7.88\n",
    "p = chi2.sf(res, 1) # 1 degrees of freedom\n",
    "print(res, 'p-value: %.30f' % p)\n",
    "#  82.37586050140558 p-value: 0.000000000000000000112519634099"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Null Hypothesis: probability of a bigram is just the product of the individual probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.50959954316171e-61"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pone = c1/N\n",
    "ptwo = c2/N\n",
    "res = binom.pmf(c12, min(c1,c2), pone * ptwo)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collocation: probability of a bigram is the conditional probability of the P(B|A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12578655488346452"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pone = c1/N\n",
    "ptwo = c2/N\n",
    "pbg = c12/N\n",
    "res2 = binom.pmf(c12, min(c1, c2), pbg / pone)\n",
    "res2\n",
    "# p(A|B) = P(AB) / P(B)\n",
    "# P(B|A) = P(AB) / P(A)\n",
    "# P(C | AB ) = P(ABC) / P(AB)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Note: Loglikelihood calculation fails with zero divide on \"of the\".  \n",
    "##### Hence, we will proactively trap a runtimeWarning: divide by zero encountered in log, which may occur with extreme collocations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "inf"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# e.g. Brown News corpus \n",
    "c1 = 2849 # of\n",
    "c2 = 5580 # the\n",
    "c12 = 844 \n",
    "N =100554\n",
    "loglikelihood_ratio(c1, c2, c12, N)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_probabilities(c1:int, c2:int, c12:int, N:int):\n",
    "    \"\"\"\n",
    "    :param c1: the count of unigram x\n",
    "    :param c2: the count of unigram y\n",
    "    :param c12: the number of times that unigrams x and y co-occur\n",
    "    :param N: the total number of words in the observed corpus\n",
    "    \n",
    "    :return tuple of probabilities:\n",
    "    (not x, not y), P(x), P(y), P(xy)\n",
    "    \"\"\"\n",
    "    zero_zero = 1 - (((c1 - c12) / N ) +( (c2 - c12) / N) + (c12 / N))\n",
    "    one_zero = (c1 - c12) / N\n",
    "    zero_one = (c2 - c12) / N\n",
    "    one_one = c12 / N\n",
    "    return zero_zero, one_zero + one_one, zero_one + one_one, one_one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PMI - Pointwise Mutual Information\n",
    "\n",
    "$pmi(x;y)\\equiv log \\frac{p(x,y)}{p(x)p(y)}$\n",
    "\n",
    "Pointwise mutual information can be normalized between [-1,+1] resulting in -1 (in the limit) for never occurring together, 0 for independence, and +1 for complete co-occurrence.\n",
    "\n",
    "$npmi(x;y) = \\frac{pmi(x;y)}{h(x,y)}$\n",
    "\n",
    "where h(x,y)\n",
    "is the joint self-information, which is estimated as \n",
    "$-\\log _{2}p(X=x,Y=y)$\n",
    "\n",
    "see also: https://en.wikipedia.org/wiki/Pointwise_mutual_information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pmi(c1, c2, c12, N):\n",
    "    \"\"\"\n",
    "    Calculate the pointwise mutual information for the word and bigram counts\n",
    "    \n",
    "    Parameters:\n",
    "    c1: count of word 1\n",
    "    c2: count of word 2\n",
    "    c12: count of bigram (w1,w2)\n",
    "    N: the number of words in the corpus\n",
    "    \n",
    "    The following example is from Wikipedia data table at:\n",
    "    https://en.wikipedia.org/wiki/Pointwise_mutual_information\n",
    "    >>> c1 = 3501  # los\n",
    "    >>> c2 = 2808 # angeles \n",
    "    >>> c12 = 2791\n",
    "    >>> N  = 50_000_952\n",
    "    >>> pmi(c1, c2, c12, N)\n",
    "    9.560676150647117\n",
    "    \n",
    "    \"\"\"    \n",
    "    p_not_x_not_y, px, py, pxy = calc_probabilities(c1, c2, c12, N)\n",
    "    return np.log(pxy / (px * py))\n",
    "\n",
    "def npmi(c1, c2, c12, N):\n",
    "    \"\"\"\n",
    "    Calculate the normalized pointwise mutual information for the word and bigram counts.\n",
    "    Normalized between [-1,+1] resulting in:\n",
    "    * -1 (in the limit) for never occurring together\n",
    "    * 0 for independence\n",
    "    * +1 for complete co-occurrence.\n",
    "    \n",
    "    Parameters:\n",
    "    c1: count of word 1\n",
    "    c2: count of word 2\n",
    "    c12: count of bigram (w1,w2)\n",
    "    N: the number of words in the corpus\n",
    "        \n",
    "    >>> c1 = 3501  # los\n",
    "    >>> c2 = 2808 # angeles \n",
    "    >>> c12 = 2791\n",
    "    >>> N  = 50_000_952\n",
    "    >>> npmi(c1, c2, c12, N)\n",
    "    0.49398779748399313\n",
    "    \n",
    "    \"\"\"    \n",
    "    p_not_x_not_y, px, py, pxy = calc_probabilities(c1, c2, c12, N)\n",
    "    pmi_val = np.log(pxy / (px * py))\n",
    "    return pmi_val / - np.log(px * py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.560676150647117"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c1 = 3501  # los\n",
    "c2 = 2808 # angeles \n",
    "c12 = 2791\n",
    "N  = 50_000_952\n",
    "pmi(c1, c2, c12, N)\n",
    "# 9.560676150647117"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.49398779748399313"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "npmi(c1, c2, c12, N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corpus Exploration\n",
    "We'll look at the Brown News Corpus, which is very small--only 100,554 tokens--and this small corpus will help show the strengths and weaknesses of our collocation algorithms.\n",
    "\n",
    "We'll add one more way of detecting/filtering collocations, by using Stanza for POS tagging. The collocation / POS tag patterns are from Justeson and Katz 1995, and their original list is: \n",
    "* A N\n",
    "* N N\n",
    "* A A N\n",
    "* A N N\n",
    "* N A N\n",
    "* N N N\n",
    "* N P N \n",
    "\n",
    "(A = Adj, N = Noun, P = Preposition).\n",
    "\n",
    "Note: Wikipedia lists a different set: There are about six main types of collocations: adjective + noun, noun + noun (such as collective nouns), verb + noun, adverb + adjective, verbs + prepositional phrase (phrasal verbs), and verb + adverb. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a76ab6e22f844e89a21f168021cbe03f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/main/resources_1.3.0.json:   0%|   …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-12-14 23:14:50 INFO: Downloading default packages for language: en (English)...\n",
      "2021-12-14 23:14:52 INFO: File exists: /Users/todd/stanza_resources/en/default.zip.\n",
      "2021-12-14 23:15:02 INFO: Finished downloading models and saved to /Users/todd/stanza_resources.\n",
      "2021-12-14 23:15:02 INFO: Loading these models for language: en (English):\n",
      "========================\n",
      "| Processor | Package  |\n",
      "------------------------\n",
      "| tokenize  | combined |\n",
      "| pos       | combined |\n",
      "========================\n",
      "\n",
      "2021-12-14 23:15:02 INFO: Use device: cpu\n",
      "2021-12-14 23:15:02 INFO: Loading: tokenize\n",
      "2021-12-14 23:15:02 INFO: Loading: pos\n",
      "2021-12-14 23:15:03 INFO: Done loading processors!\n"
     ]
    }
   ],
   "source": [
    "stanza.download('en')\n",
    "nlp = stanza.Pipeline(lang='en', processors='tokenize,pos',\n",
    "                      use_gpu=False, tokenize_batch_size=1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4623/4623 [05:02<00:00, 15.30it/s]\n"
     ]
    }
   ],
   "source": [
    "word_counter = Counter() # brown.words(categories='news'))\n",
    "# we wish to filter out punctuation, so this isn't enough:\n",
    "# bigrams = [list(ngrams(tmp, 2)) for tmp in brown.sents(categories='news')]\n",
    "bigrams = []\n",
    "trigrams = []\n",
    "words_upos = {}\n",
    "\n",
    "COLL_POS_TG = {('ADJ', 'ADJ', 'NOUN'),\n",
    "                ('ADJ', 'NOUN', 'NOUN'), \n",
    "                ('NOUN', 'ADJ', 'NOUN'),               \n",
    "                ('NOUN', 'NOUN', 'NOUN'), \n",
    "                ('NOUN', 'ADP', 'NOUN'),                \n",
    "                ('PROPN', 'PROPN', 'PROPN')}\n",
    "\n",
    "COLL_POS_BG = {('ADJ', 'NOUN'), # these are the top 3 required\n",
    "                ('NOUN', 'NOUN'), \n",
    "                ('PROPN', 'PROPN'),                \n",
    "                ('ADJ', 'ADJ'),\n",
    "                # The following are derived from the trigram POS tags, \n",
    "                # and we have to capture them to compute probabilities               \n",
    "                ('NOUN', 'ADP'),                \n",
    "                ('ADP', 'NOUN'),                               \n",
    "                ('NOUN', 'ADJ')}\n",
    "\n",
    "for sent in tqdm(brown.sents(categories='news'),\n",
    "                 total=len(brown.sents(categories='news'))):\n",
    "    clean_sent = []\n",
    "    clean_sent_upos = []\n",
    "    # This will be a coarse grain mapping of POS tags\n",
    "    doc = nlp(' '.join(sent))\n",
    "    for sentence in doc.sentences:\n",
    "        for word in sentence.words:\n",
    "            words_upos[word.text] = word.upos\n",
    "            if word.text.isalpha():\n",
    "                clean_sent.append(word.text)    \n",
    "                word_counter.update({word.text: 1})\n",
    "                clean_sent_upos.append(word.upos)\n",
    "    candidate_bigrams = list(ngrams(clean_sent, 2))\n",
    "    candidate_trigrams = list(ngrams(clean_sent, 3))\n",
    "    for idx, can in enumerate(list(ngrams(clean_sent_upos, 2))):\n",
    "        if can in COLL_POS_BG:            \n",
    "            bigrams.append(candidate_bigrams[idx])\n",
    "    for idx, can in enumerate(list(ngrams(clean_sent_upos, 3))):\n",
    "        if can in COLL_POS_TG:            \n",
    "            trigrams.append(candidate_trigrams[idx])\n",
    "\n",
    "bigram_counter = Counter()\n",
    "for bg in bigrams:\n",
    "    bigram_counter.update({bg: 1})\n",
    "    \n",
    "trigram_counter = Counter()\n",
    "for tg in trigrams:\n",
    "    trigram_counter.update({tg:1})\n",
    "\n",
    "# del bigrams, trigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('one', 'of')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "candidate_bigrams[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('National', 'Library', 'Week')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "candidate_trigrams[8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('per', 'cent'), 50),\n",
       " (('last', 'year'), 33),\n",
       " (('members', 'of'), 32),\n",
       " (('last', 'week'), 30),\n",
       " (('president', 'of'), 28),\n",
       " (('number', 'of'), 27),\n",
       " (('President', 'Kennedy'), 25),\n",
       " (('chairman', 'of'), 22),\n",
       " (('home', 'runs'), 22),\n",
       " (('part', 'of'), 19)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_counter.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('toll', 'road', 'bonds'), 8),\n",
       " (('home', 'rule', 'charter'), 5),\n",
       " (('years', 'in', 'prison'), 4),\n",
       " (('aged', 'care', 'plan'), 4),\n",
       " (('grants', 'to', 'states'), 4),\n",
       " (('board', 'of', 'directors'), 4),\n",
       " (('La', 'Dolce', 'Vita'), 4),\n",
       " (('potato', 'chip', 'industry'), 4),\n",
       " (('Catholic', 'higher', 'education'), 4),\n",
       " (('discrimination', 'in', 'employment'), 3)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trigram_counter.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accurately estimating trigrams and bigrams\n",
    "we don't have to create and store all the bigrams and trigrams to estimate their totals, as long as we know the number of sentences in the corpus, and the number of unigrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4623 18747 3697 85270\n"
     ]
    }
   ],
   "source": [
    "total_sents = len(brown.sents(categories='news'))\n",
    "total_bigrams = sum(bigram_counter.values())\n",
    "total_trigrams = sum(trigram_counter.values())\n",
    "total_unigrams = sum(word_counter.values())\n",
    "print(total_sents, total_bigrams, total_trigrams, total_unigrams)\n",
    "# (4623, 81396, 76832, 86002)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4.301861631194324, 20.563700297538546)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "est_bigrams = total_unigrams - total_sents\n",
    "est_trigrams = total_unigrams - 2*(total_sents)\n",
    "est_bigrams / total_bigrams, est_trigrams / total_trigrams\n",
    "# (0.9997911445279867, 0.9990108288213244)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15038, 3550)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(bigram_counter), len(trigram_counter)\n",
    "# (54578, 71695)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('toll', 'road', 'bonds'), 8),\n",
       " (('home', 'rule', 'charter'), 5),\n",
       " (('years', 'in', 'prison'), 4),\n",
       " (('aged', 'care', 'plan'), 4),\n",
       " (('grants', 'to', 'states'), 4),\n",
       " (('board', 'of', 'directors'), 4),\n",
       " (('La', 'Dolce', 'Vita'), 4),\n",
       " (('potato', 'chip', 'industry'), 4),\n",
       " (('Catholic', 'higher', 'education'), 4),\n",
       " (('discrimination', 'in', 'employment'), 3),\n",
       " (('Speaker', 'Sam', 'Rayburn'), 3),\n",
       " (('South', 'Viet', 'Nam'), 3),\n",
       " (('Prince', 'Souvanna', 'Phouma'), 3),\n",
       " (('civil', 'defense', 'program'), 3),\n",
       " (('fire', 'fighters', 'association'), 3)]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trigram_counter.most_common(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# N = sum(bigram_counter.values())\n",
    "N = sum(word_counter.values())\n",
    "data = []\n",
    "for bigram in bigram_counter.keys():\n",
    "    w1, w2 = bigram\n",
    "    c1 = word_counter[w1]\n",
    "    c2 = word_counter[w2]        \n",
    "    npmi_val = npmi(c1, c2, bigram_counter[bigram], N)\n",
    "    loglikelihood_ratio_val = -2 * loglikelihood_ratio(c1, c2, bigram_counter[bigram], N)\n",
    "    data.append({\"w1\": w1,\n",
    "                    \"w2\": w2,\n",
    "                    \"w1w2_pos\": f\"{words_upos[w1]} {words_upos[w2]}\",\n",
    "                    \"npmi\": npmi_val,\n",
    "                    \"loglikelihood_ratio\": loglikelihood_ratio_val,\n",
    "                    \"c1\": c1,\n",
    "                    \"c2\": c2,                \n",
    "                    \"bigram_cnt\": bigram_counter[bigram],\n",
    "                    \"w1pos\": words_upos[w1],                 \n",
    "                    \"w2pos\": words_upos[w2]})\n",
    "df = pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sorting by loglikelihood_ratio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w1w2_pos</th>\n",
       "      <th>npmi</th>\n",
       "      <th>loglikelihood_ratio</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>bigram_cnt</th>\n",
       "      <th>w1pos</th>\n",
       "      <th>w2pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2517</th>\n",
       "      <td>professional</td>\n",
       "      <td>posts</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.432809</td>\n",
       "      <td>29.904074</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13754</th>\n",
       "      <td>next</td>\n",
       "      <td>fall</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.303898</td>\n",
       "      <td>8.019506</td>\n",
       "      <td>40</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>491</th>\n",
       "      <td>state</td>\n",
       "      <td>deaf</td>\n",
       "      <td>NOUN ADJ</td>\n",
       "      <td>0.312862</td>\n",
       "      <td>8.573055</td>\n",
       "      <td>97</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14762</th>\n",
       "      <td>cooperative</td>\n",
       "      <td>movement</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.406307</td>\n",
       "      <td>14.004512</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14165</th>\n",
       "      <td>Byron</td>\n",
       "      <td>Rickards</td>\n",
       "      <td>PROPN PROPN</td>\n",
       "      <td>0.430458</td>\n",
       "      <td>15.709993</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11567</th>\n",
       "      <td>High</td>\n",
       "      <td>blood</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.395293</td>\n",
       "      <td>13.282395</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>580</th>\n",
       "      <td>utility</td>\n",
       "      <td>companies</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.376429</td>\n",
       "      <td>11.980586</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14109</th>\n",
       "      <td>Government</td>\n",
       "      <td>Defense</td>\n",
       "      <td>PROPN PROPN</td>\n",
       "      <td>0.367919</td>\n",
       "      <td>11.478318</td>\n",
       "      <td>23</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5699</th>\n",
       "      <td>Thursday</td>\n",
       "      <td>June</td>\n",
       "      <td>PROPN PROPN</td>\n",
       "      <td>0.327795</td>\n",
       "      <td>9.192198</td>\n",
       "      <td>21</td>\n",
       "      <td>16</td>\n",
       "      <td>1</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14152</th>\n",
       "      <td>age</td>\n",
       "      <td>boy</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.342881</td>\n",
       "      <td>9.992428</td>\n",
       "      <td>12</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 w1         w2     w1w2_pos      npmi  loglikelihood_ratio  \\\n",
       "2517   professional      posts     ADJ NOUN  0.432809            29.904074   \n",
       "13754          next       fall     ADJ NOUN  0.303898             8.019506   \n",
       "491           state       deaf     NOUN ADJ  0.312862             8.573055   \n",
       "14762   cooperative   movement     ADJ NOUN  0.406307            14.004512   \n",
       "14165         Byron   Rickards  PROPN PROPN  0.430458            15.709993   \n",
       "11567          High      blood     ADJ NOUN  0.395293            13.282395   \n",
       "580         utility  companies    NOUN NOUN  0.376429            11.980586   \n",
       "14109    Government    Defense  PROPN PROPN  0.367919            11.478318   \n",
       "5699       Thursday       June  PROPN PROPN  0.327795             9.192198   \n",
       "14152           age        boy    NOUN NOUN  0.342881             9.992428   \n",
       "\n",
       "       c1  c2  bigram_cnt  w1pos  w2pos  \n",
       "2517   10   5           2    ADJ   NOUN  \n",
       "13754  40  15           1    ADJ   NOUN  \n",
       "491    97   5           1   NOUN    ADJ  \n",
       "14762   3  12           1    ADJ   NOUN  \n",
       "14165   4   4           1  PROPN  PROPN  \n",
       "11567  17   3           1    ADJ   NOUN  \n",
       "580     5  18           1   NOUN   NOUN  \n",
       "14109  23   5           1  PROPN  PROPN  \n",
       "5699   21  16           1  PROPN  PROPN  \n",
       "14152  12  19           1   NOUN   NOUN  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for a alpha value of: 0.005, 1 Degree of freedom requires a chi-squared value of 7.88\n",
    "df.query('loglikelihood_ratio > 7.88 and c1 >2 and c2 >2').sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sorting by NPMI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w1w2_pos</th>\n",
       "      <th>npmi</th>\n",
       "      <th>loglikelihood_ratio</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>bigram_cnt</th>\n",
       "      <th>w1pos</th>\n",
       "      <th>w2pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>establishment</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.253343</td>\n",
       "      <td>35.120483</td>\n",
       "      <td>7</td>\n",
       "      <td>2850</td>\n",
       "      <td>6</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4058</th>\n",
       "      <td>on</td>\n",
       "      <td>completions</td>\n",
       "      <td>ADP NOUN</td>\n",
       "      <td>0.321732</td>\n",
       "      <td>29.199831</td>\n",
       "      <td>658</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>ADP</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9397</th>\n",
       "      <td>beginning</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.199217</td>\n",
       "      <td>11.105044</td>\n",
       "      <td>7</td>\n",
       "      <td>2850</td>\n",
       "      <td>3</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6052</th>\n",
       "      <td>trip</td>\n",
       "      <td>to</td>\n",
       "      <td>NOUN PART</td>\n",
       "      <td>0.195250</td>\n",
       "      <td>10.291986</td>\n",
       "      <td>10</td>\n",
       "      <td>2125</td>\n",
       "      <td>3</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>PART</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1316</th>\n",
       "      <td>East</td>\n",
       "      <td>West</td>\n",
       "      <td>PROPN ADJ</td>\n",
       "      <td>0.390679</td>\n",
       "      <td>44.560445</td>\n",
       "      <td>22</td>\n",
       "      <td>26</td>\n",
       "      <td>4</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4015</th>\n",
       "      <td>Last</td>\n",
       "      <td>year</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.304879</td>\n",
       "      <td>21.692448</td>\n",
       "      <td>16</td>\n",
       "      <td>178</td>\n",
       "      <td>3</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8121</th>\n",
       "      <td>in</td>\n",
       "      <td>sales</td>\n",
       "      <td>ADP NOUN</td>\n",
       "      <td>0.110732</td>\n",
       "      <td>4.407956</td>\n",
       "      <td>1896</td>\n",
       "      <td>52</td>\n",
       "      <td>4</td>\n",
       "      <td>ADP</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2571</th>\n",
       "      <td>head</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.212545</td>\n",
       "      <td>34.441728</td>\n",
       "      <td>26</td>\n",
       "      <td>2850</td>\n",
       "      <td>10</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1492</th>\n",
       "      <td>Souvanna</td>\n",
       "      <td>Phouma</td>\n",
       "      <td>PROPN PROPN</td>\n",
       "      <td>0.494340</td>\n",
       "      <td>82.734057</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2923</th>\n",
       "      <td>performance</td>\n",
       "      <td>bond</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.427460</td>\n",
       "      <td>41.749275</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 w1           w2     w1w2_pos      npmi  loglikelihood_ratio  \\\n",
       "124   establishment           of     NOUN ADP  0.253343            35.120483   \n",
       "4058             on  completions     ADP NOUN  0.321732            29.199831   \n",
       "9397      beginning           of     NOUN ADP  0.199217            11.105044   \n",
       "6052           trip           to    NOUN PART  0.195250            10.291986   \n",
       "1316           East         West    PROPN ADJ  0.390679            44.560445   \n",
       "4015           Last         year     ADJ NOUN  0.304879            21.692448   \n",
       "8121             in        sales     ADP NOUN  0.110732             4.407956   \n",
       "2571           head           of     NOUN ADP  0.212545            34.441728   \n",
       "1492       Souvanna       Phouma  PROPN PROPN  0.494340            82.734057   \n",
       "2923    performance         bond    NOUN NOUN  0.427460            41.749275   \n",
       "\n",
       "        c1    c2  bigram_cnt  w1pos  w2pos  \n",
       "124      7  2850           6   NOUN    ADP  \n",
       "4058   658     3           3    ADP   NOUN  \n",
       "9397     7  2850           3   NOUN    ADP  \n",
       "6052    10  2125           3   NOUN   PART  \n",
       "1316    22    26           4  PROPN    ADJ  \n",
       "4015    16   178           3    ADJ   NOUN  \n",
       "8121  1896    52           4    ADP   NOUN  \n",
       "2571    26  2850          10   NOUN    ADP  \n",
       "1492     5     4           4  PROPN  PROPN  \n",
       "2923    11    11           3   NOUN   NOUN  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# it's commonly recommended to filter PMI values by unigrams with at least 3 occurences\n",
    "df.query('c1 > 2 and c2 > 2 and bigram_cnt > 2 and npmi > 0').sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sorting by POS tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w1w2_pos</th>\n",
       "      <th>npmi</th>\n",
       "      <th>loglikelihood_ratio</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>bigram_cnt</th>\n",
       "      <th>w1pos</th>\n",
       "      <th>w2pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fulton</td>\n",
       "      <td>County</td>\n",
       "      <td>PROPN PROPN</td>\n",
       "      <td>0.419954</td>\n",
       "      <td>75.185197</td>\n",
       "      <td>14</td>\n",
       "      <td>36</td>\n",
       "      <td>6</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>investigation</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.206101</td>\n",
       "      <td>15.167637</td>\n",
       "      <td>9</td>\n",
       "      <td>2850</td>\n",
       "      <td>4</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>recent</td>\n",
       "      <td>primary</td>\n",
       "      <td>ADJ ADJ</td>\n",
       "      <td>0.327323</td>\n",
       "      <td>9.167252</td>\n",
       "      <td>20</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>primary</td>\n",
       "      <td>election</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.343422</td>\n",
       "      <td>18.668591</td>\n",
       "      <td>17</td>\n",
       "      <td>38</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>in</td>\n",
       "      <td>presentments</td>\n",
       "      <td>ADP NOUN</td>\n",
       "      <td>0.251066</td>\n",
       "      <td>7.612668</td>\n",
       "      <td>1896</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>ADP</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>charge</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.221895</td>\n",
       "      <td>31.499885</td>\n",
       "      <td>17</td>\n",
       "      <td>2850</td>\n",
       "      <td>8</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>thanks</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.123974</td>\n",
       "      <td>1.730451</td>\n",
       "      <td>6</td>\n",
       "      <td>2850</td>\n",
       "      <td>1</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>manner</td>\n",
       "      <td>in</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.193232</td>\n",
       "      <td>7.075228</td>\n",
       "      <td>7</td>\n",
       "      <td>1896</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>September</td>\n",
       "      <td>October</td>\n",
       "      <td>PROPN PROPN</td>\n",
       "      <td>0.376429</td>\n",
       "      <td>11.928196</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>term</td>\n",
       "      <td>jury</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.290850</td>\n",
       "      <td>7.409629</td>\n",
       "      <td>18</td>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              w1            w2     w1w2_pos      npmi  loglikelihood_ratio  \\\n",
       "0         Fulton        County  PROPN PROPN  0.419954            75.185197   \n",
       "1  investigation            of     NOUN ADP  0.206101            15.167637   \n",
       "2         recent       primary      ADJ ADJ  0.327323             9.167252   \n",
       "3        primary      election     ADJ NOUN  0.343422            18.668591   \n",
       "4             in  presentments     ADP NOUN  0.251066             7.612668   \n",
       "5         charge            of     NOUN ADP  0.221895            31.499885   \n",
       "6         thanks            of     NOUN ADP  0.123974             1.730451   \n",
       "7         manner            in     NOUN ADP  0.193232             7.075228   \n",
       "8      September       October  PROPN PROPN  0.376429            11.928196   \n",
       "9           term          jury    NOUN NOUN  0.290850             7.409629   \n",
       "\n",
       "     c1    c2  bigram_cnt  w1pos  w2pos  \n",
       "0    14    36           6  PROPN  PROPN  \n",
       "1     9  2850           4   NOUN    ADP  \n",
       "2    20    17           1    ADJ    ADJ  \n",
       "3    17    38           2    ADJ   NOUN  \n",
       "4  1896     1           1    ADP   NOUN  \n",
       "5    17  2850           8   NOUN    ADP  \n",
       "6     6  2850           1   NOUN    ADP  \n",
       "7     7  1896           2   NOUN    ADP  \n",
       "8    10     9           1  PROPN  PROPN  \n",
       "9    18    45           1   NOUN   NOUN  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query(\"w1w2_pos in ['NOUN NOUN', 'ADJ NOUN', 'PROPN PROPN', 'NOUN ADJ', \" \n",
    "         \" 'NOUN ADP', 'ADJ ADJ','ADP NOUN']\").head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sorting by loglikelihood_ratio, and POS tags (without Proper Nouns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w1w2_pos</th>\n",
       "      <th>npmi</th>\n",
       "      <th>loglikelihood_ratio</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>bigram_cnt</th>\n",
       "      <th>w1pos</th>\n",
       "      <th>w2pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7905</th>\n",
       "      <td>magnetic</td>\n",
       "      <td>tape</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.492887</td>\n",
       "      <td>63.031007</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>211</th>\n",
       "      <td>per</td>\n",
       "      <td>cent</td>\n",
       "      <td>ADP NOUN</td>\n",
       "      <td>0.491872</td>\n",
       "      <td>769.601616</td>\n",
       "      <td>61</td>\n",
       "      <td>52</td>\n",
       "      <td>50</td>\n",
       "      <td>ADP</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9055</th>\n",
       "      <td>potato</td>\n",
       "      <td>chip</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.490466</td>\n",
       "      <td>96.627662</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8046</th>\n",
       "      <td>collective</td>\n",
       "      <td>bargaining</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.489033</td>\n",
       "      <td>162.815779</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7896</th>\n",
       "      <td>data</td>\n",
       "      <td>processing</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.485558</td>\n",
       "      <td>78.177368</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4580</th>\n",
       "      <td>relative</td>\n",
       "      <td>merits</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.480231</td>\n",
       "      <td>39.003530</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9906</th>\n",
       "      <td>depletion</td>\n",
       "      <td>allowances</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.480231</td>\n",
       "      <td>39.003530</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7612</th>\n",
       "      <td>utility</td>\n",
       "      <td>pole</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.473788</td>\n",
       "      <td>54.069549</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3659</th>\n",
       "      <td>mound</td>\n",
       "      <td>chores</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.472837</td>\n",
       "      <td>37.277461</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7903</th>\n",
       "      <td>electronic</td>\n",
       "      <td>data</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.472837</td>\n",
       "      <td>37.277461</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              w1          w2   w1w2_pos      npmi  loglikelihood_ratio  c1  \\\n",
       "7905    magnetic        tape   ADJ NOUN  0.492887            63.031007   3   \n",
       "211          per        cent   ADP NOUN  0.491872           769.601616  61   \n",
       "9055      potato        chip  NOUN NOUN  0.490466            96.627662   6   \n",
       "8046  collective  bargaining   ADJ NOUN  0.489033           162.815779  10   \n",
       "7896        data  processing  NOUN NOUN  0.485558            78.177368   4   \n",
       "4580    relative      merits   ADJ NOUN  0.480231            39.003530   3   \n",
       "9906   depletion  allowances  NOUN NOUN  0.480231            39.003530   3   \n",
       "7612     utility        pole  NOUN NOUN  0.473788            54.069549   5   \n",
       "3659       mound      chores  NOUN NOUN  0.472837            37.277461   3   \n",
       "7903  electronic        data   ADJ NOUN  0.472837            37.277461   3   \n",
       "\n",
       "      c2  bigram_cnt w1pos w2pos  \n",
       "7905   4           3   ADJ  NOUN  \n",
       "211   52          50   ADP  NOUN  \n",
       "9055   6           5  NOUN  NOUN  \n",
       "8046  12           9   ADJ  NOUN  \n",
       "7896   7           4  NOUN  NOUN  \n",
       "4580   3           2   ADJ  NOUN  \n",
       "9906   3           2  NOUN  NOUN  \n",
       "7612   5           3  NOUN  NOUN  \n",
       "3659   4           2  NOUN  NOUN  \n",
       "7903   4           2   ADJ  NOUN  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for a alpha value of: 0.005, 1 Degree of freedom requires a chi-squared value of 7.88\n",
    "df.query(\"loglikelihood_ratio > 7.88 \" \n",
    "         \" and c1 > 2 and c2 > 2 and w1w2_pos in \" \n",
    "         \" ['NOUN NOUN', 'ADJ NOUN', 'NOUN ADJ', 'NOUN ADP', 'ADJ ADJ', 'ADP NOUN']\"\n",
    "        ).sort_values(by=['npmi', 'loglikelihood_ratio'], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w1w2_pos</th>\n",
       "      <th>npmi</th>\n",
       "      <th>loglikelihood_ratio</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>bigram_cnt</th>\n",
       "      <th>w1pos</th>\n",
       "      <th>w2pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>211</th>\n",
       "      <td>per</td>\n",
       "      <td>cent</td>\n",
       "      <td>ADP NOUN</td>\n",
       "      <td>0.491872</td>\n",
       "      <td>769.601616</td>\n",
       "      <td>61</td>\n",
       "      <td>52</td>\n",
       "      <td>50</td>\n",
       "      <td>ADP</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1820</th>\n",
       "      <td>last</td>\n",
       "      <td>week</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.392631</td>\n",
       "      <td>265.530790</td>\n",
       "      <td>161</td>\n",
       "      <td>93</td>\n",
       "      <td>30</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>977</th>\n",
       "      <td>last</td>\n",
       "      <td>year</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.368604</td>\n",
       "      <td>250.974446</td>\n",
       "      <td>161</td>\n",
       "      <td>178</td>\n",
       "      <td>33</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8046</th>\n",
       "      <td>collective</td>\n",
       "      <td>bargaining</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.489033</td>\n",
       "      <td>162.815779</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>number</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.286063</td>\n",
       "      <td>156.366511</td>\n",
       "      <td>32</td>\n",
       "      <td>2850</td>\n",
       "      <td>27</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1763</th>\n",
       "      <td>vice</td>\n",
       "      <td>president</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.445414</td>\n",
       "      <td>142.567836</td>\n",
       "      <td>11</td>\n",
       "      <td>54</td>\n",
       "      <td>10</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1640</th>\n",
       "      <td>last</td>\n",
       "      <td>night</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.366580</td>\n",
       "      <td>140.568827</td>\n",
       "      <td>161</td>\n",
       "      <td>65</td>\n",
       "      <td>17</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>members</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.249033</td>\n",
       "      <td>123.882829</td>\n",
       "      <td>70</td>\n",
       "      <td>2850</td>\n",
       "      <td>32</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>952</th>\n",
       "      <td>social</td>\n",
       "      <td>security</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.464201</td>\n",
       "      <td>123.555102</td>\n",
       "      <td>17</td>\n",
       "      <td>13</td>\n",
       "      <td>8</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>688</th>\n",
       "      <td>high</td>\n",
       "      <td>school</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.390520</td>\n",
       "      <td>117.732428</td>\n",
       "      <td>52</td>\n",
       "      <td>67</td>\n",
       "      <td>12</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              w1          w2   w1w2_pos      npmi  loglikelihood_ratio   c1  \\\n",
       "211          per        cent   ADP NOUN  0.491872           769.601616   61   \n",
       "1820        last        week   ADJ NOUN  0.392631           265.530790  161   \n",
       "977         last        year   ADJ NOUN  0.368604           250.974446  161   \n",
       "8046  collective  bargaining   ADJ NOUN  0.489033           162.815779   10   \n",
       "22        number          of   NOUN ADP  0.286063           156.366511   32   \n",
       "1763        vice   president  NOUN NOUN  0.445414           142.567836   11   \n",
       "1640        last       night   ADJ NOUN  0.366580           140.568827  161   \n",
       "320      members          of   NOUN ADP  0.249033           123.882829   70   \n",
       "952       social    security   ADJ NOUN  0.464201           123.555102   17   \n",
       "688         high      school   ADJ NOUN  0.390520           117.732428   52   \n",
       "\n",
       "        c2  bigram_cnt w1pos w2pos  \n",
       "211     52          50   ADP  NOUN  \n",
       "1820    93          30   ADJ  NOUN  \n",
       "977    178          33   ADJ  NOUN  \n",
       "8046    12           9   ADJ  NOUN  \n",
       "22    2850          27  NOUN   ADP  \n",
       "1763    54          10  NOUN  NOUN  \n",
       "1640    65          17   ADJ  NOUN  \n",
       "320   2850          32  NOUN   ADP  \n",
       "952     13           8   ADJ  NOUN  \n",
       "688     67          12   ADJ  NOUN  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query(\"loglikelihood_ratio > 7.88 \" \n",
    "         \" and c1 >2 and c2 >2 and w1w2_pos in \" \n",
    "         \" ['NOUN NOUN', 'ADJ NOUN', 'NOUN ADJ', 'NOUN ADP', 'ADJ ADJ', 'ADP NOUN']\"\n",
    "        ).sort_values(by=['loglikelihood_ratio'], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "85270"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "loglikefilter = len(df.query('loglikelihood_ratio > 7.88 and c1 >2 and c2 >2'))\n",
    "loglike_pos_filter = len(df.query(\"loglikelihood_ratio > 7.88 and c1 >2 and c2 >2 and w1w2_pos in ['NOUN NOUN', 'ADJ NOUN', 'NOUN ADJ', 'NOUN ADP', 'ADJ ADJ','ADP NOUN']\").sort_values(by=['npmi', 'loglikelihood_ratio'], ascending=False))\n",
    "pos_filter = len(df.query(\"w1w2_pos in ['NOUN NOUN', 'ADJ NOUN', 'PROPN PROPN', 'NOUN ADJ', 'NOUN ADP', 'ADJ ADJ','ADP NOUN']\"))\n",
    "npmi_filter = len(df.query('c1 > 2 and c2 > 2 and bigram_cnt > 2 and npmi > 0'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAfpElEQVR4nO3deZgdVb3u8e9LwiBjgLQcSJAgiUNARYyA1+HkGC4EUIP3iIIDAXKMeFFxOhCc4DAIHkEEZTgIEYIIRBSIAmJEIpd7mILkMkWkCWASGRoyADIZ+N0/1mpSafbu7vTu7r26+/08z35StWrVqlW1a9e7a+gdRQRmZmalWafZHTAzM6vFAWVmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAWZ+TdIKkJyU9VkBfDpZ0U2U8JI3t5z6cI+nbzViepImSlvTDMu+VNLGvl9PbJF0g6YQ+antAbpNmckD1M0mflDRf0rOSHpV0raT3NbtffUXSG4CvAeMj4p96qU1J+pKkeyT9XdISSb+Q9LbeaL83dQxEgIg4LCKO74NlHSvpH3nfan8d2dnyJD0saY8Gl/uag3pE7BgR8xppt86y5uUvFe/oUH5FLp/Y28vsLX21TQYzB1Q/kvRV4IfAd4GtgDcAZwFT+rEPktSf7/sbgKci4om1nVHS8DqTTgeOAL4EbAG8CbgS2LeHfRxMLouIjSuv/2yksU7eg2b6C3BQ+4ikLYH3AG1N65H1jYjwqx9ewGbAs8D+ndRZnxRgf8uvHwLr52kLgQ9V6g4nfSB3yeO7A/8NrAD+HzCxUncecCLwf4HngbHAIbnNZ4BFwOc69OVI4NHcj38DAhhb6ecpwF+Bx4FzgNfVWJ898vJeyet+QS7/CHBv7us84K2VeR4GjgLuAl4EhndocxzwMrBrF9t6Vt4+jwDfAtbJ0w4GbqrUra5X3fny9M9Wttl9lW0/A3iwUv7RXP5W4IXc32eBFbn8AuCEDu22AsuAOcA2Hfp3GPBA3l5nAqqz3scCP6tR/urygInAkjx8UX5vns/9OxIYk5c5Lb+/N+a6vwAeA1YCNwI75vLpwD+Al3Ibv668j3t0Y7+eCCwhnWU/QdrnDunkvZ0HfCfPMyyXfQE4O5dNzGXrVN6Xp4DZwBaVdt7H6s/LYuDgyrY6E7g6v5+3AjtU5js9138auAN4f4ftP5u0Dz1D2scndNi327fJrsD83M7jwA9yefv2PyQvZ3l+/99N+kysAH5caXMs8Mf8vjxJ+oLS9ONdb72a3oGh8gImA6vocMDtUOc44Bbg9UBL/gAdn6d9B7i4UndfYGEeHpU/hPvkD+b/zOMtefo80sFmR1KwrZvn3wEQ8M/Ac6w+4E4mHYx2BDYEfsaaB/LTSAfSLYBNgF8DJ9VZp4nkA2IefxPw99zHdUkHxVZgvTz9YWABsC21Q+8w4JEutvUs4KrctzGkb9zT8rSDqR9Qnc23P7A0HyiUDwzbVaZtk7f9J/L6bV1rebnsAlYHxgfzgWUX0oH8R+RQqPTvN8AI0tloGzC5znofy1oEVGV771EZH5OXOQvYqP09AA7N26U9bBbUar9Wu3S+X08kfS6Oy/vDPqR9cfM66ziP9IXpd8Deuew20hlUNaCOyMscnfv8X8Aledp2pAA5MC9zS2Dnyro8RQqQ4cDFwKWV5X861x9OCtXHgA0q2/+FvA7DgJOAW+psk5uBz+ThjYHdO2z/c4ANgD1zm1fm7TeKFOT/nOtfAnyTtO9tALyv2ce63nw1vQND5QV8CnisizoPAvtUxvcCHs7DY/OHasM8fjHwnTx8FHBRh7auA6bm4XnAcV0s+0rgiDw8k0rg5GVH/lekA3D1W+V7gIfqtDuRNQ+I3wZmV8bXIR34J+bxh4FDO+nnN6sf+hrTh5G+zY+vlH0OmJeHD6ZGQHVjvuvat0833usFwJRay8tlF7A6MM4H/rMybWPSGcmYSv/eV5k+G5hRZ7nH5nVYUXltQ88C6o2drN+IXGezjutTq106368nks7ghlemP0E+YNdY9jxSQH2adHB+C/CXPK0aUAuBSZX5ts7bdThwNHBFnfYvAM6rjO8D/LmTbbEceEdl+/++Mm088HydbXIj8B/AyA7ttW//UZWyp4BPVMZ/CXw5D88CzgVGd2ffHGgv34PqP08BI7u4pr8N6dJSu0dyGRHRSvrQfVjShqTLZD/P9bYD9pe0ov1FuoSxdaWtxdUFSdpb0i2SluX6+wAjK/1YXGfeFtJZ1R2VZf02l3fHGusYEa/k9kfV62sHT7HmenU0kvStuON2HFW7erfn25Z0oH0NSQdJWlDZHjuxelt2peP2eJa0jtX+Vp9+fI4UYvXMjogRldffutmPjl59DyQNk3SypAclPU060EIP15HKfp09FRGrKuNdrSPAr0hnn18gXarsaDvgisp7spB0qXUrOnkvs7rbW9LXJS2UtDK3uxlrboeO825Q5zM/jXQ14c+Sbpf0oQ7TH68MP19jvL1PR5K+NN6WnxI8tJP1GnAcUP3nZtI9lf06qfM30ger3RtyWbtLSJclpgD35dCCdDC5qMOBaaOIOLkyb7QPSFqf9C3sFGCriBgBXEPa0SHdBxhdmXfbyvCTpA/IjpVlbRYRXR1Qaq6jJOX2l9bqaw3XA6MlTagz/UnSN+WO23Fp7erdnm8x6ZLoGiRtB/yEdKDcMm/Le1i9LTtbF3jt9tiIdAmpq/72lnr9q5Z/krTP7UE6II/J5T1aR167X6+1iHgOuBb4PLUDajHpEmD1M7FBRCylznvZFUnvJwXCx0mXIEeQ7v2os/nq9P+BiDiQdNnue8Dl+b1f23Yei4jPRsQ2pDP+s/r7zyb6kgOqn0TEStJ9pDMl7SdpQ0nr5jOZ9ietLgG+JalF0shc/2eVZi4lXZP+PKvPnsh1Pixpr/xtd4P89y7VkKlaj3Rdvg1YJWnv3G672cAhkt6az9Ze/ZudfMbzE+A0Sa8HkDRK0l7d3BSzgX0lTZK0Luk6/ouk+xJdiogHSE8+XpLXcb28vgdImhERL+dlnChpkxwgX2XN7Vir3a7mOw/4uqR35Schx+Y6G5EO0G15WxxCOoNq9zgpUNers+hLSNt65/zF4bvArRHxcHe2Ry94HHhjF3U2Ib1HT5HOnr+7lm10tV/31DdI92IerjHtHNJ7uR1AXvaUPO1iYA9JH5c0XNKWknbuxvI2Id0vawOGS/oOsGlPOi7p05Ja8udpRS5+pQft7F/5nC8n7Ytr3U6pHFD9KCJOJR30vkXayReTvnlfmaucQHqy5y7gbuBPuax9/kdJZ2L/A7isUr6Y9A33G5V2/506729EPEN6RHs2aaf+JOmhh/bp1wJnADeQHmC4JU96Mf97VHt5vuTze+DN3dwG95PuH/yIdNbyYeDDEfFSd+bPvgT8mPS01QrS5ZqPkh7WAPgi6T7ZIuAmUpjP7Ea7deeLiF+QnoT8Oele4JWkp8LuA04lvS+PA28jPS3Z7g+kp7kek/RkxwVGxO9JXwB+STpz3QE4oBt97S0nkcJjhaSv16kzi3RZbinpKcVbOkw/Hxif27iyxvyd7tc9FRF/i4ib6kw+nbRP/07SM7nPu+X5/kq6pP010pOTC4B3dGOR15EuZ/+FtD1eoPPL0Z2ZDNwr6dnc1wMi4vketPNu4NbczhzSfdJFPexTcZRvtJnVJemtpMtW63e4V2Bm1md8BmU1SfqopPUlbU66Rv5rh5OZ9ScHlNXzOdLjvg+Snn76fHO7Y2ZDjS/xmZlZkXwGZWZmRSrxhyC7ZeTIkTFmzJhmd8PMzBp0xx13PBkRr/lj/wEbUGPGjGH+/PnN7oaZmTVI0iO1yn2Jz8zMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyIN2J86MhsKxsy4utldGFQePnnfZnfB1oLPoMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSF0GlKSZkp6QdE+l7PuS/izpLklXSBpRmXa0pFZJ90vaq1I+OZe1SppRKd9e0q25/DJJ6/Xi+pmZ2QDVnTOoC4DJHcrmAjtFxNuBvwBHA0gaDxwA7JjnOUvSMEnDgDOBvYHxwIG5LsD3gNMiYiywHJjW0BqZmdmg0GVARcSNwLIOZb+LiFV59BZgdB6eAlwaES9GxENAK7BrfrVGxKKIeAm4FJgiScAHgcvz/BcC+zW2SmZmNhj0xj2oQ4Fr8/AoYHFl2pJcVq98S2BFJezay2uSNF3SfEnz29raeqHrZmZWqoYCStI3gVXAxb3Tnc5FxLkRMSEiJrS0tPTHIs3MrEmG93RGSQcDHwImRUTk4qXAtpVqo3MZdcqfAkZIGp7Poqr1zcxsCOvRGZSkycCRwEci4rnKpDnAAZLWl7Q9MA64DbgdGJef2FuP9CDFnBxsNwAfy/NPBa7q2aqYmdlg0p3HzC8BbgbeLGmJpGnAj4FNgLmSFkg6ByAi7gVmA/cBvwUOj4iX89nRF4DrgIXA7FwX4Cjgq5JaSfekzu/VNTQzswGpy0t8EXFgjeK6IRIRJwIn1ii/BrimRvki0lN+ZmZmr/IvSZiZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZG6DChJMyU9IemeStkWkuZKeiD/u3kul6QzJLVKukvSLpV5pub6D0iaWil/l6S78zxnSFJvr6SZmQ083TmDugCY3KFsBnB9RIwDrs/jAHsD4/JrOnA2pEADjgF2A3YFjmkPtVzns5X5Oi7LzMyGoC4DKiJuBJZ1KJ4CXJiHLwT2q5TPiuQWYISkrYG9gLkRsSwilgNzgcl52qYRcUtEBDCr0paZmQ1hPb0HtVVEPJqHHwO2ysOjgMWVektyWWflS2qU1yRpuqT5kua3tbX1sOtmZjYQNPyQRD7ziV7oS3eWdW5ETIiICS0tLf2xSDMza5KeBtTj+fIc+d8ncvlSYNtKvdG5rLPy0TXKzcxsiOtpQM0B2p/EmwpcVSk/KD/NtzuwMl8KvA7YU9Lm+eGIPYHr8rSnJe2en947qNKWmZkNYcO7qiDpEmAiMFLSEtLTeCcDsyVNAx4BPp6rXwPsA7QCzwGHAETEMknHA7fnesdFRPuDF/+b9KTg64Br88vMzIa4LgMqIg6sM2lSjboBHF6nnZnAzBrl84GduuqHmZkNLf4lCTMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgNBZSkr0i6V9I9ki6RtIGk7SXdKqlV0mWS1st118/jrXn6mEo7R+fy+yXt1eA6mZnZINDjgJI0CvgSMCEidgKGAQcA3wNOi4ixwHJgWp5lGrA8l5+W6yFpfJ5vR2AycJakYT3tl5mZDQ6NXuIbDrxO0nBgQ+BR4IPA5Xn6hcB+eXhKHidPnyRJufzSiHgxIh4CWoFdG+yXmZkNcD0OqIhYCpwC/JUUTCuBO4AVEbEqV1sCjMrDo4DFed5Vuf6W1fIa86xB0nRJ8yXNb2tr62nXzcxsAGjkEt/mpLOf7YFtgI1Il+j6TEScGxETImJCS0tLXy7KzMyarJFLfHsAD0VEW0T8A/gV8F5gRL7kBzAaWJqHlwLbAuTpmwFPVctrzGNmZkNUIwH1V2B3SRvme0mTgPuAG4CP5TpTgavy8Jw8Tp7+h4iIXH5Afspve2AccFsD/TIzs0FgeNdVaouIWyVdDvwJWAXcCZwLXA1cKumEXHZ+nuV84CJJrcAy0pN7RMS9kmaTwm0VcHhEvNzTfpmZ2eDQ44ACiIhjgGM6FC+ixlN4EfECsH+ddk4ETmykL2ZmNrj4lyTMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiNRRQkkZIulzSnyUtlPQeSVtImivpgfzv5rmuJJ0hqVXSXZJ2qbQzNdd/QNLURlfKzMwGvkbPoE4HfhsRbwHeASwEZgDXR8Q44Po8DrA3MC6/pgNnA0jaAjgG2A3YFTimPdTMzGzo6nFASdoM+ABwPkBEvBQRK4ApwIW52oXAfnl4CjArkluAEZK2BvYC5kbEsohYDswFJve0X2ZmNjg0cga1PdAG/FTSnZLOk7QRsFVEPJrrPAZslYdHAYsr8y/JZfXKzcxsCGskoIYDuwBnR8Q7gb+z+nIeABERQDSwjDVImi5pvqT5bW1tvdWsmZkVqJGAWgIsiYhb8/jlpMB6PF+6I//7RJ6+FNi2Mv/oXFav/DUi4tyImBARE1paWhroupmZla7HARURjwGLJb05F00C7gPmAO1P4k0FrsrDc4CD8tN8uwMr86XA64A9JW2eH47YM5eZmdkQNrzB+b8IXCxpPWARcAgp9GZLmgY8Anw8170G2AdoBZ7LdYmIZZKOB27P9Y6LiGUN9svMzAa4hgIqIhYAE2pMmlSjbgCH12lnJjCzkb6Ymdng4l+SMDOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI1+lNHZmYDxpgZVze7C4POwyfv22dt+wzKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDQeUpGGS7pT0mzy+vaRbJbVKukzSerl8/TzemqePqbRxdC6/X9JejfbJzMwGvt44gzoCWFgZ/x5wWkSMBZYD03L5NGB5Lj8t10PSeOAAYEdgMnCWpGG90C8zMxvAGgooSaOBfYHz8riADwKX5yoXAvvl4Sl5nDx9Uq4/Bbg0Il6MiIeAVmDXRvplZmYDX6NnUD8EjgReyeNbAisiYlUeXwKMysOjgMUAefrKXP/V8hrzrEHSdEnzJc1va2trsOtmZlayHgeUpA8BT0TEHb3Yn05FxLkRMSEiJrS0tPTXYs3MrAmGNzDve4GPSNoH2ADYFDgdGCFpeD5LGg0szfWXAtsCSyQNBzYDnqqUt6vOY2ZmQ1SPz6Ai4uiIGB0RY0gPOfwhIj4F3AB8LFebClyVh+fkcfL0P0RE5PID8lN+2wPjgNt62i8zMxscGjmDquco4FJJJwB3Aufn8vOBiyS1AstIoUZE3CtpNnAfsAo4PCJe7oN+mZnZANIrARUR84B5eXgRNZ7Ci4gXgP3rzH8icGJv9MXMzAYH/5KEmZkVyQFlZmZF6ot7UDZEjZlxdbO7YGaDiM+gzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEgOKDMzK5IDyszMiuSAMjOzIjmgzMysSA4oMzMrkgPKzMyK5IAyM7MiOaDMzKxIDigzMyuSA8rMzIrkgDIzsyI5oMzMrEg9DihJ20q6QdJ9ku6VdEQu30LSXEkP5H83z+WSdIakVkl3Sdql0tbUXP8BSVMbXy0zMxvoGjmDWgV8LSLGA7sDh0saD8wAro+IccD1eRxgb2Bcfk0HzoYUaMAxwG7ArsAx7aFmZmZDV48DKiIejYg/5eFngIXAKGAKcGGudiGwXx6eAsyK5BZghKStgb2AuRGxLCKWA3OByT3tl5mZDQ69cg9K0hjgncCtwFYR8Wie9BiwVR4eBSyuzLYkl9Urr7Wc6ZLmS5rf1tbWG103M7NCNRxQkjYGfgl8OSKerk6LiACi0WVU2js3IiZExISWlpbeatbMzArUUEBJWpcUThdHxK9y8eP50h353ydy+VJg28rso3NZvXIzMxvCGnmKT8D5wMKI+EFl0hyg/Um8qcBVlfKD8tN8uwMr86XA64A9JW2eH47YM5eZmdkQNryBed8LfAa4W9KCXPYN4GRgtqRpwCPAx/O0a4B9gFbgOeAQgIhYJul44PZc77iIWNZAv8zMbBDocUBFxE2A6kyeVKN+AIfXaWsmMLOnfTEzs8HHvyRhZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmViQHlJmZFckBZWZmRXJAmZlZkRxQZmZWJAeUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRWpkf9Rd8AbM+PqZnfBzMzq8BmUmZkVyQFlZmZFckCZmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZEcUGZmVqRiAkrSZEn3S2qVNKPZ/TEzs+YqIqAkDQPOBPYGxgMHShrf3F6ZmVkzFRFQwK5Aa0QsioiXgEuBKU3uk5mZNVEpv8U3ClhcGV8C7NaxkqTpwPQ8+qyk+/uhbyUYCTzZ7E4MIt6evc/btHcNmO2p7/VKM9vVKiwloLolIs4Fzm12P/qbpPkRMaHZ/RgsvD17n7dp7/L2TEq5xLcU2LYyPjqXmZnZEFVKQN0OjJO0vaT1gAOAOU3uk5mZNVERl/giYpWkLwDXAcOAmRFxb5O7VZIhd1mzj3l79j5v097l7QkoIprdBzMzs9co5RKfmZnZGhxQZmZWJAdUE0gKSadWxr8u6dg8fKykpZIWSLpH0kcq5SFpbGW+L+eyCXn8YUkj+3l11oqkZ3uhjYmSfpOHP9L+01iSLpD0sR62uYGkK/M2v1PSGzup+7CkuyXdJel3kv4pl28maVb+ua4H8/Bmedo6ks7I7d8t6XZJ2/ekr/1N0suV/fEXkjbM5aMlXSXpgby+p+eHnJC0oaSL87reI+kmSRs3d036R2/v4x3Kz2v/lZ3eWE7pHFDN8SLwvzoJk9MiYmdgf2CmpPb36W7SE47t9geG9MMkETEnIk7uhab2B1ZGxE7AB4FlXdT/l4h4OzAf+EYuOx9YFBFjI2IH4CHgvDztE8A2wNsj4m3AR4EVvdDv/vB8ROyct81LwGGSBPwKuDIixgFvAjYGTszzHAE8HhFvy/NNA/7RhL4PKhHxbxFxX7P70V8cUM2xivSUzlc6qxQRC3Pd9iC7kvwTUJJ2AFYyQP7avDOSdpZ0Sz4juULS5rn83blsgaTvS7qnxrwHS/pxjfLj8xnVMEn/ns9Y7pL0H3W68RIwSpIiYnlErOhm928ExuYz23cBx1emHQdMyO/V1sCjEfEKQEQsiYjl3VxGSf4PMJYU4i9ExE8BIuJl0v58aD7D2prK3zJGxP0R8WIT+luERvbxDu3Ma79iUikbKelmSftKapH0y7y/3y7pvX25Xn3NAdU8ZwKfar8EVIuk3YBXgLZc9DSwWNJOpDOpy/q8l/1jFnBUPiO5Gzgml/8U+Fw+m3y5u41J+j7QAhwCTALGkX7vcWfgXZI+UGO2RcAuwElr2fcP5T6PBxbkAzXw6kF7AbAjMBv4cD4QnSrpnWu5nKaTNJz0g853k9bpjur0iHga+CspwGYCR+UD5wmSxvV3fwvTq/t4O0lbAVcD34mIq4HTSVdg3g38K6vP4AckB1ST5A/zLOBLNSZ/RdIC4BTgE7Hm3wJcSgqn/YAr+ribfS4H9IiI+GMuuhD4gKQRwCYRcXMu/3k3m/w2sFlEHJa32575dSfwJ+AtpMCq9uF1pAPFm4GdJX05l1+dvwzUckN+jzalG6EWEUty+0eTvnRcL2lSN9ep2V6X13U+KYDO72qGiFgAvBH4PrAFcLukt/ZhH4vVB/t4u3WB64EjI2JuLtsD+HF+v+YAmw7ke39F/KHuEPZD0kHzpx3KT4uIU+rM8xvSh35+RDydbgVYxe2ks6QtImIZIOCkiPivTuZ5G/BkRLRJ+lfg95JeIR1Y693j+5eIePXyqqT7SOG2TvtlvHzvcGfgPoB8ieta4FpJj5O+ZFzf81XtN8/nb/ivyuv7sQ5lmwJvAFoBIuJZ0n2qX+XtuQ+wsD86PESsIp3F7gW0h986wO4R8ULTetWLfAbVRPkAOpt0A7m78zwHHMXqm9EDWkSsBJZLen8u+gzwx3wP6Jl8mRPWfDikM78FTgaulrQJ6ddJDm3/FilplKTXd5jnAeAtknaMiL+T3o9TgKs6nL12th6tpLO0b1WKvwX8KSJaJe0iaZvch3WAtwOPdHOdSnQ9sKGkg+DV/9PtVOCCiHhO0nsr91nWI10CHcjr22N9sI+/2jRwKGnfPSqX/Q74YnsFSTv3tN8l8BlU850KfGFtZoiIS/uoL/1hQ0lLKuM/AKYC5+Sb64tI944gBcVP8rfvP5IeCulSRPwih9Mc0rf2nwM357PNZ4FPA09U6i+XNBW4KD+dthL4FHCSpBsj4r+7uW7TgB9JejCP38zqLx+vz+uyfh6/DXjNwx0DRUSEpI8CZ0n6NunL7jWsfqJxB+DsvD3XId0n+WVTOtv/emsfn9Shnf07LigiXpZ0IDBH0jOkWwZnSrqLdHy/ETisl9ar3/mnjqxYkjbOl4lQ+lunrSPiiCZ3y6zXeB/vnM+grGT7SjqatJ8+Ahzc3O6Y9Trv453wGZSZmRXJD0mYmVmRHFBmZlYkB5SZmRXJAWVmZkVyQJmZWZH+PwRi0wVhPsx3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "heights = [npmi_filter, loglike_pos_filter, pos_filter, loglikefilter]\n",
    "labels = [\"NPMI\", \"Loglike & POS\", \"POS\", \"LogLike\"]\n",
    "fig, ax = plt.subplots()\n",
    "x = np.arange(len(labels)) \n",
    "width = 1\n",
    "ax.bar(x, heights, width)\n",
    "ax.set_title('Coverage for Collocation Filtration Mechanisms')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter by loglike_pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w1w2_pos</th>\n",
       "      <th>npmi</th>\n",
       "      <th>loglikelihood_ratio</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>bigram_cnt</th>\n",
       "      <th>w1pos</th>\n",
       "      <th>w2pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>investigation</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.206101</td>\n",
       "      <td>15.167637</td>\n",
       "      <td>9</td>\n",
       "      <td>2850</td>\n",
       "      <td>4</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>recent</td>\n",
       "      <td>primary</td>\n",
       "      <td>ADJ ADJ</td>\n",
       "      <td>0.327323</td>\n",
       "      <td>9.167252</td>\n",
       "      <td>20</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>primary</td>\n",
       "      <td>election</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.343422</td>\n",
       "      <td>18.668591</td>\n",
       "      <td>17</td>\n",
       "      <td>38</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>charge</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.221895</td>\n",
       "      <td>31.499885</td>\n",
       "      <td>17</td>\n",
       "      <td>2850</td>\n",
       "      <td>8</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>possible</td>\n",
       "      <td>irregularities</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.378783</td>\n",
       "      <td>12.261064</td>\n",
       "      <td>28</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15016</th>\n",
       "      <td>inadequate</td>\n",
       "      <td>service</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.358915</td>\n",
       "      <td>11.016435</td>\n",
       "      <td>4</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15018</th>\n",
       "      <td>college</td>\n",
       "      <td>libraries</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.321191</td>\n",
       "      <td>8.855464</td>\n",
       "      <td>18</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15031</th>\n",
       "      <td>public</td>\n",
       "      <td>understanding</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.338597</td>\n",
       "      <td>9.863899</td>\n",
       "      <td>51</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15032</th>\n",
       "      <td>main</td>\n",
       "      <td>reasons</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.397239</td>\n",
       "      <td>13.325174</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15035</th>\n",
       "      <td>theme</td>\n",
       "      <td>For</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.363575</td>\n",
       "      <td>11.228237</td>\n",
       "      <td>5</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3500 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  w1              w2   w1w2_pos      npmi  \\\n",
       "1      investigation              of   NOUN ADP  0.206101   \n",
       "2             recent         primary    ADJ ADJ  0.327323   \n",
       "3            primary        election   ADJ NOUN  0.343422   \n",
       "5             charge              of   NOUN ADP  0.221895   \n",
       "14          possible  irregularities   ADJ NOUN  0.378783   \n",
       "...              ...             ...        ...       ...   \n",
       "15016     inadequate         service   ADJ NOUN  0.358915   \n",
       "15018        college       libraries  NOUN NOUN  0.321191   \n",
       "15031         public   understanding   ADJ NOUN  0.338597   \n",
       "15032           main         reasons   ADJ NOUN  0.397239   \n",
       "15035          theme             For   NOUN ADP  0.363575   \n",
       "\n",
       "       loglikelihood_ratio  c1    c2  bigram_cnt w1pos w2pos  \n",
       "1                15.167637   9  2850           4  NOUN   ADP  \n",
       "2                 9.167252  20    17           1   ADJ   ADJ  \n",
       "3                18.668591  17    38           2   ADJ  NOUN  \n",
       "5                31.499885  17  2850           8  NOUN   ADP  \n",
       "14               12.261064  28     3           1   ADJ  NOUN  \n",
       "...                    ...  ..   ...         ...   ...   ...  \n",
       "15016            11.016435   4    37           1   ADJ  NOUN  \n",
       "15018             8.855464  18    22           1  NOUN  NOUN  \n",
       "15031             9.863899  51     5           1   ADJ  NOUN  \n",
       "15032            13.325174   4    12           1   ADJ  NOUN  \n",
       "15035            11.228237   5    26           1  NOUN   ADP  \n",
       "\n",
       "[3500 rows x 10 columns]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loglike_pos_df = df.query(\"loglikelihood_ratio > 7.88 and \" \n",
    "                          \" c1 >2 and c2 >2 and w1w2_pos in ['NOUN NOUN', 'ADJ NOUN', \"\n",
    "                          \" 'NOUN ADJ', 'NOUN ADP', 'ADJ ADJ','ADP NOUN']\")\n",
    "loglike_pos_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering by NPMI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w1w2_pos</th>\n",
       "      <th>npmi</th>\n",
       "      <th>loglikelihood_ratio</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>bigram_cnt</th>\n",
       "      <th>w1pos</th>\n",
       "      <th>w2pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fulton</td>\n",
       "      <td>County</td>\n",
       "      <td>PROPN PROPN</td>\n",
       "      <td>0.419954</td>\n",
       "      <td>75.185197</td>\n",
       "      <td>14</td>\n",
       "      <td>36</td>\n",
       "      <td>6</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>investigation</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.206101</td>\n",
       "      <td>15.167637</td>\n",
       "      <td>9</td>\n",
       "      <td>2850</td>\n",
       "      <td>4</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>charge</td>\n",
       "      <td>of</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.221895</td>\n",
       "      <td>31.499885</td>\n",
       "      <td>17</td>\n",
       "      <td>2850</td>\n",
       "      <td>8</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>reports</td>\n",
       "      <td>of</td>\n",
       "      <td>VERB ADP</td>\n",
       "      <td>0.187483</td>\n",
       "      <td>12.460307</td>\n",
       "      <td>12</td>\n",
       "      <td>2850</td>\n",
       "      <td>4</td>\n",
       "      <td>VERB</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>interest</td>\n",
       "      <td>in</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.193385</td>\n",
       "      <td>20.369824</td>\n",
       "      <td>33</td>\n",
       "      <td>1896</td>\n",
       "      <td>7</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14720</th>\n",
       "      <td>York</td>\n",
       "      <td>State</td>\n",
       "      <td>PROPN PROPN</td>\n",
       "      <td>0.296402</td>\n",
       "      <td>20.307167</td>\n",
       "      <td>54</td>\n",
       "      <td>63</td>\n",
       "      <td>3</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>PROPN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14728</th>\n",
       "      <td>libraries</td>\n",
       "      <td>in</td>\n",
       "      <td>NOUN ADP</td>\n",
       "      <td>0.192605</td>\n",
       "      <td>15.254045</td>\n",
       "      <td>22</td>\n",
       "      <td>1896</td>\n",
       "      <td>5</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14902</th>\n",
       "      <td>in</td>\n",
       "      <td>member</td>\n",
       "      <td>ADP NOUN</td>\n",
       "      <td>0.114129</td>\n",
       "      <td>3.670967</td>\n",
       "      <td>1896</td>\n",
       "      <td>36</td>\n",
       "      <td>3</td>\n",
       "      <td>ADP</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14919</th>\n",
       "      <td>member</td>\n",
       "      <td>libraries</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.378311</td>\n",
       "      <td>41.773555</td>\n",
       "      <td>36</td>\n",
       "      <td>22</td>\n",
       "      <td>4</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14922</th>\n",
       "      <td>library</td>\n",
       "      <td>service</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.345243</td>\n",
       "      <td>27.016210</td>\n",
       "      <td>31</td>\n",
       "      <td>37</td>\n",
       "      <td>3</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>677 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  w1         w2     w1w2_pos      npmi  loglikelihood_ratio  \\\n",
       "0             Fulton     County  PROPN PROPN  0.419954            75.185197   \n",
       "1      investigation         of     NOUN ADP  0.206101            15.167637   \n",
       "5             charge         of     NOUN ADP  0.221895            31.499885   \n",
       "13           reports         of     VERB ADP  0.187483            12.460307   \n",
       "21          interest         in     NOUN ADP  0.193385            20.369824   \n",
       "...              ...        ...          ...       ...                  ...   \n",
       "14720           York      State  PROPN PROPN  0.296402            20.307167   \n",
       "14728      libraries         in     NOUN ADP  0.192605            15.254045   \n",
       "14902             in     member     ADP NOUN  0.114129             3.670967   \n",
       "14919         member  libraries    NOUN NOUN  0.378311            41.773555   \n",
       "14922        library    service    NOUN NOUN  0.345243            27.016210   \n",
       "\n",
       "         c1    c2  bigram_cnt  w1pos  w2pos  \n",
       "0        14    36           6  PROPN  PROPN  \n",
       "1         9  2850           4   NOUN    ADP  \n",
       "5        17  2850           8   NOUN    ADP  \n",
       "13       12  2850           4   VERB    ADP  \n",
       "21       33  1896           7   NOUN    ADP  \n",
       "...     ...   ...         ...    ...    ...  \n",
       "14720    54    63           3  PROPN  PROPN  \n",
       "14728    22  1896           5   NOUN    ADP  \n",
       "14902  1896    36           3    ADP   NOUN  \n",
       "14919    36    22           4   NOUN   NOUN  \n",
       "14922    31    37           3   NOUN   NOUN  \n",
       "\n",
       "[677 rows x 10 columns]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "npmi_df = df.query('c1 > 2 and c2 > 2 and bigram_cnt > 2 and npmi > 0')\n",
    "npmi_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loglikelihood ratio and NPMI only overlap by 0.11%\n"
     ]
    }
   ],
   "source": [
    "llpidx = loglike_pos_df.index.tolist()\n",
    "npmiidx = npmi_df.index.tolist()\n",
    "overlap = len(set(llpidx) & set(npmiidx))\n",
    "percent_overlap = overlap / max(len(loglike_pos_df), len(npmi_df))\n",
    "print(f\"Loglikelihood ratio and NPMI only overlap by {round(percent_overlap, 2)}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter all, using: npmi, loglikelihood ratio, POS tags\n",
    "for a alpha value of: 0.005, 1 Degree of freedom requires a chi-squared value of 7.88"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w1w2_pos</th>\n",
       "      <th>npmi</th>\n",
       "      <th>loglikelihood_ratio</th>\n",
       "      <th>c1</th>\n",
       "      <th>c2</th>\n",
       "      <th>bigram_cnt</th>\n",
       "      <th>w1pos</th>\n",
       "      <th>w2pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7905</th>\n",
       "      <td>magnetic</td>\n",
       "      <td>tape</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.492887</td>\n",
       "      <td>63.031007</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>211</th>\n",
       "      <td>per</td>\n",
       "      <td>cent</td>\n",
       "      <td>ADP NOUN</td>\n",
       "      <td>0.491872</td>\n",
       "      <td>769.601616</td>\n",
       "      <td>61</td>\n",
       "      <td>52</td>\n",
       "      <td>50</td>\n",
       "      <td>ADP</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9055</th>\n",
       "      <td>potato</td>\n",
       "      <td>chip</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.490466</td>\n",
       "      <td>96.627662</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8046</th>\n",
       "      <td>collective</td>\n",
       "      <td>bargaining</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.489033</td>\n",
       "      <td>162.815779</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7896</th>\n",
       "      <td>data</td>\n",
       "      <td>processing</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.485558</td>\n",
       "      <td>78.177368</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4580</th>\n",
       "      <td>relative</td>\n",
       "      <td>merits</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.480231</td>\n",
       "      <td>39.003530</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9906</th>\n",
       "      <td>depletion</td>\n",
       "      <td>allowances</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.480231</td>\n",
       "      <td>39.003530</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7612</th>\n",
       "      <td>utility</td>\n",
       "      <td>pole</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.473788</td>\n",
       "      <td>54.069549</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3659</th>\n",
       "      <td>mound</td>\n",
       "      <td>chores</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.472837</td>\n",
       "      <td>37.277461</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7903</th>\n",
       "      <td>electronic</td>\n",
       "      <td>data</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.472837</td>\n",
       "      <td>37.277461</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8632</th>\n",
       "      <td>aerial</td>\n",
       "      <td>spraying</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.472837</td>\n",
       "      <td>37.277461</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9913</th>\n",
       "      <td>raw</td>\n",
       "      <td>materials</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.472837</td>\n",
       "      <td>37.277461</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12665</th>\n",
       "      <td>nineteenth</td>\n",
       "      <td>century</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.472837</td>\n",
       "      <td>37.277461</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769</th>\n",
       "      <td>blue</td>\n",
       "      <td>ribbon</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.467037</td>\n",
       "      <td>52.446113</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5336</th>\n",
       "      <td>buffet</td>\n",
       "      <td>supper</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.466955</td>\n",
       "      <td>36.092545</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6852</th>\n",
       "      <td>spy</td>\n",
       "      <td>ring</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.466955</td>\n",
       "      <td>36.092545</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10407</th>\n",
       "      <td>acacia</td>\n",
       "      <td>panels</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.466955</td>\n",
       "      <td>36.092545</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3643</th>\n",
       "      <td>wild</td>\n",
       "      <td>pitch</td>\n",
       "      <td>ADJ NOUN</td>\n",
       "      <td>0.466955</td>\n",
       "      <td>36.092545</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3775</th>\n",
       "      <td>pounds</td>\n",
       "      <td>lighter</td>\n",
       "      <td>NOUN ADJ</td>\n",
       "      <td>0.466955</td>\n",
       "      <td>36.092545</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>ADJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9712</th>\n",
       "      <td>corn</td>\n",
       "      <td>pickers</td>\n",
       "      <td>NOUN NOUN</td>\n",
       "      <td>0.466955</td>\n",
       "      <td>36.092545</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NOUN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               w1          w2   w1w2_pos      npmi  loglikelihood_ratio  c1  \\\n",
       "7905     magnetic        tape   ADJ NOUN  0.492887            63.031007   3   \n",
       "211           per        cent   ADP NOUN  0.491872           769.601616  61   \n",
       "9055       potato        chip  NOUN NOUN  0.490466            96.627662   6   \n",
       "8046   collective  bargaining   ADJ NOUN  0.489033           162.815779  10   \n",
       "7896         data  processing  NOUN NOUN  0.485558            78.177368   4   \n",
       "4580     relative      merits   ADJ NOUN  0.480231            39.003530   3   \n",
       "9906    depletion  allowances  NOUN NOUN  0.480231            39.003530   3   \n",
       "7612      utility        pole  NOUN NOUN  0.473788            54.069549   5   \n",
       "3659        mound      chores  NOUN NOUN  0.472837            37.277461   3   \n",
       "7903   electronic        data   ADJ NOUN  0.472837            37.277461   3   \n",
       "8632       aerial    spraying   ADJ NOUN  0.472837            37.277461   3   \n",
       "9913          raw   materials   ADJ NOUN  0.472837            37.277461   4   \n",
       "12665  nineteenth     century   ADJ NOUN  0.472837            37.277461   3   \n",
       "769          blue      ribbon   ADJ NOUN  0.467037            52.446113   8   \n",
       "5336       buffet      supper  NOUN NOUN  0.466955            36.092545   3   \n",
       "6852          spy        ring  NOUN NOUN  0.466955            36.092545   3   \n",
       "10407      acacia      panels  NOUN NOUN  0.466955            36.092545   3   \n",
       "3643         wild       pitch   ADJ NOUN  0.466955            36.092545   5   \n",
       "3775       pounds     lighter   NOUN ADJ  0.466955            36.092545   5   \n",
       "9712         corn     pickers  NOUN NOUN  0.466955            36.092545   5   \n",
       "\n",
       "       c2  bigram_cnt w1pos w2pos  \n",
       "7905    4           3   ADJ  NOUN  \n",
       "211    52          50   ADP  NOUN  \n",
       "9055    6           5  NOUN  NOUN  \n",
       "8046   12           9   ADJ  NOUN  \n",
       "7896    7           4  NOUN  NOUN  \n",
       "4580    3           2   ADJ  NOUN  \n",
       "9906    3           2  NOUN  NOUN  \n",
       "7612    5           3  NOUN  NOUN  \n",
       "3659    4           2  NOUN  NOUN  \n",
       "7903    4           2   ADJ  NOUN  \n",
       "8632    4           2   ADJ  NOUN  \n",
       "9913    3           2   ADJ  NOUN  \n",
       "12665   4           2   ADJ  NOUN  \n",
       "769     4           3   ADJ  NOUN  \n",
       "5336    5           2  NOUN  NOUN  \n",
       "6852    5           2  NOUN  NOUN  \n",
       "10407   5           2  NOUN  NOUN  \n",
       "3643    3           2   ADJ  NOUN  \n",
       "3775    3           2  NOUN   ADJ  \n",
       "9712    3           2  NOUN  NOUN  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query(\"npmi > 0 and \" \n",
    "         \"loglikelihood_ratio > 7.88 and \" \n",
    "         \"c1 > 2 and c2 > 2 and w1w2_pos in \"\n",
    "         \"['NOUN NOUN', 'ADJ NOUN', 'NOUN ADJ', 'NOUN ADP', 'ADJ ADJ','ADP NOUN']\"\n",
    "        ).sort_values(by=['npmi', 'loglikelihood_ratio'], ascending=False).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "* Some POS tagging provide false signals: `PROPN PROPN` yielding \"September October\"\n",
    "* highest quality collocations are found by ensemble filtration of:\n",
    "    * NPMI\n",
    "    * POS tags\n",
    "    * Loglikelihood ratio\n",
    "    * minimum threshold for the bigram occurences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MLYCU 3.9",
   "language": "python",
   "name": "mlycu3.9"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}