probabilistic_language_modeling/make_trie_language_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making a Trie Language Model\n",
    "## A Trie Language Model is an efficient data structure that organizes a good, large baseline sample of known correct words in the target language.\n",
    "### Whenever possible it can be helpful to have Trie Language Model to assist you in diagnosing a corpus and potentially fixing problems or auto-correcting bad data. \n",
    "### Prior to starting, we have examined some of the CLTK data structures and found a source of some known correct words that we will use to build our tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### First some standard imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import site\n",
    "import pickle\n",
    "import random\n",
    "import os\n",
    "from collections import defaultdict, Counter\n",
    "from tqdm import tqdm\n",
    "\n",
    "from cltk.stem.latin.j_v import JVReplacer\n",
    "from cltk.corpus.readers import get_corpus_reader\n",
    "from cltk.tokenize.word import WordTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add parent directory to path so we can access our common code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, sys, inspect\n",
    "currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))\n",
    "parentdir = os.path.dirname(currentdir)\n",
    "sys.path.insert(0, parentdir) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlyoucanuse.word_trie import WordTrie\n",
    "from mlyoucanuse.aeoe_replacer import AEOEReplacer\n",
    "from mlyoucanuse.corpus_analysis_fun import get_split_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The CLTK Lemmata package is outside of the regular CLTK library path, but we would like to work with the Lemmmata dictionary to access a large collection of good inflected Latin words so we will add the directory to our Python path programmatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of distinct root word forms: 239,321\n"
     ]
    }
   ],
   "source": [
    "site.addsitedir(os.path.expanduser('~/cltk_data/latin/lemma/latin_pos_lemmata_cltk'))\n",
    "# Now we may reference the dictionary in code\n",
    "from latin_unambiguous_lemmata_cltk import LEMMATA\n",
    "print(f'Number of distinct root word forms: {len(LEMMATA.keys()):,}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's take examine the character content of the Lemmata values to make sure there's no odd characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 239321/239321 [00:04<00:00, 53293.90it/s] 4824/239321 [00:00<00:04, 48223.60it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'e': 258882, 'i': 237115, 'a': 205356, 't': 178453, 'u': 175885, 'r': 174307, 's': 163438, 'n': 148734, 'o': 113542, 'm': 93943, 'c': 92743, 'l': 68276, 'd': 67361, 'p': 61526, 'b': 39004, 'q': 27267, 'g': 26816, 'v': 23259, 'f': 21091, 'h': 13181, 'x': 11709, 'y': 4720, 'A': 3471, 'C': 3174, 'P': 2811, 'S': 2162, 'T': 1804, 'M': 1740, 'L': 1361, 'H': 1103, 'B': 872, 'E': 867, 'I': 834, 'D': 817, 'N': 806, 'V': 708, 'G': 703, 'O': 623, 'F': 569, 'z': 409, 'R': 405, 'U': 258, 'Q': 104, 'Z': 93, 'X': 50, 'K': 21, 'k': 5, '-': 3})\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "chars = Counter()\n",
    "for word in tqdm(LEMMATA.keys()):\n",
    "    for letter in word:\n",
    "        chars.update({letter: 1})\n",
    "print(chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### There are no Greek or odd characters. However, there is the letter V, so we'll normalize the forms (in Latin J is the same as I and V is the same as U) so we'll coerce forms with the CLTK JVReplacer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 239321/239321 [00:01<00:00, 123688.41it/s]10220/239321 [00:00<00:02, 102196.69it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max length in known lemma corpus: 28 for: Thensaurochrysonicochrysides\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Thensaurochrysonicochrysides',\n",
       " 'honorificentissimisque',\n",
       " 'incomprehensibilemque',\n",
       " 'incomprehensibilisque',\n",
       " 'reconciliationibusque']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jv_replacer = JVReplacer()\n",
    "distinct_words =[jv_replacer.replace(word) for word in tqdm(LEMMATA.keys())]\n",
    "distinct_words.sort(key=len, reverse=True)\n",
    "print(f'Max length in known lemma corpus: {len(distinct_words[0])} for: {distinct_words[0]}')\n",
    "distinct_words[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The prominence of the -que enclitic ending indicates we strip off those enclitics. We can do this by tokenizing the words and only taking the first token:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['incomprehensibilem', '-que']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toker = WordTokenizer(language='latin')\n",
    "toker.tokenize('incomprehensibilemque')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 239321/239321 [00:33<00:00, 7088.98it/s]| 561/239321 [00:00<00:42, 5604.10it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Absolute maximum length 28\n",
      "Greatest common maximum length 18\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "distinct_tokenized = [toker.tokenize(tmp)[0] for tmp in tqdm(distinct_words)]\n",
    "print(f'Absolute maximum length {len(distinct_tokenized[0])}')\n",
    "print(f'Greatest common maximum length {len(distinct_tokenized[2])}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Based on our previous word length analyses, we can see that any word over a length of 18 is likely an accidentally joined word, usually created by improper formatting or a botched data import. In practice it is common to use a lower cutoff, as the 98th percentile starts with words 12 letters long.  Adjusting the cutoff is a much like the tuning of a hyperparameter: the proper value depends on the data and your needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's build the WordTrie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 239321/239321 [00:01<00:00, 168323.04it/s]5529/239321 [00:00<00:04, 52597.54it/s]\n"
     ]
    }
   ],
   "source": [
    "latin_trie = WordTrie()\n",
    "for word in tqdm(distinct_tokenized):\n",
    "    latin_trie.add(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's show how the WordTrie can be used to split improperly joined words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['maturitatem', 'perueniunt']\n",
      "['radicibus', 'subministres']\n",
      "['peregrinationes', 'habere']\n",
      "['uersibus', 'disertissimis']\n",
      "['crudelitatis', 'consuetudinem']\n",
      "['adiciebat', 'contrahendam']\n",
      "['translationes', 'inprobas']\n"
     ]
    }
   ],
   "source": [
    "bads =[\n",
    "['maturitatemperueniunt'],\n",
    "['radicibussubministres'],\n",
    "['peregrinationeshabere'],\n",
    "['uersibusdisertissimis'],\n",
    "['crudelitatisconsuetudinem'],\n",
    "['adiciebatcontrahendam'],\n",
    "['translationesinprobas'],\n",
    "]\n",
    "for item in bads:\n",
    "    print(latin_trie.extract_word_pair(item[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the word trie on a large corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2141/2141 [19:11<00:00,  1.60files/s]    | 1/2141 [00:00<07:02,  5.07files/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "229 files with oddly joined words out of 2141 corpus files\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('1644.txt',\n",
       "  [['captiuorum', 'quispiam'],\n",
       "   ['quomodo', 'comparantur'],\n",
       "   ['ingrediuntur', 'concubinae'],\n",
       "   ['tauros', 'postulauere'],\n",
       "   ['astantibus', 'manifestat'],\n",
       "   ['pluribus', 'necasset'],\n",
       "   ['latera', 'confodiunt'],\n",
       "   ['arbores', 'radicitus'],\n",
       "   ['frondibus', 'eiusdem'],\n",
       "   ['nauigabiles', 'habet'],\n",
       "   ['ceterum', 'breuissimo'],\n",
       "   ['uocalem', 'pronuntiant'],\n",
       "   ['fidelium', 'sacerdos'],\n",
       "   ['sepulcris', 'infundunt'],\n",
       "   ['perdiderat', 'quidam'],\n",
       "   ['Senam', 'propagandae'],\n",
       "   ['inseruiebat', 'aegris'],\n",
       "   ['confessiones', 'excipit'],\n",
       "   ['salutationem', 'exhibent'],\n",
       "   ['aliquando', 'continet']]),\n",
       " ('abelard/dialogus.txt', [['circumcide', 'rentur']]),\n",
       " ('alanus/alanus1.txt',\n",
       "  [['contra', 'positionem'],\n",
       "   ['prae', 'conceptionis'],\n",
       "   ['inter', 'familiaritatis']]),\n",
       " ('albertanus/albertanus.arsloquendi.txt',\n",
       "  [['passionibus', 'alienus'],\n",
       "   ['mendacio', 'redimere'],\n",
       "   ['mendacium', 'penitus'],\n",
       "   ['psalterium', 'suauem'],\n",
       "   ['iniuriam', 'cohibere'],\n",
       "   ['propulsant', 'iniuriam'],\n",
       "   ['eius', 'expositionem'],\n",
       "   ['potuit', 'prosequendo'],\n",
       "   ['doctrinam', 'eloquii'],\n",
       "   ['expositionem', 'huius'],\n",
       "   ['casibus', 'implicatur'],\n",
       "   ['uerborum', 'dignitas'],\n",
       "   ['tantam', 'uelocitatem'],\n",
       "   ['effundas', 'sermonem'],\n",
       "   ['antequam', 'loquaris'],\n",
       "   ['similibus', 'negotiis'],\n",
       "   ['salutationem', 'dicens'],\n",
       "   ['praeposuit', 'archangelus'],\n",
       "   ['uoluntatis', 'arbitrio'],\n",
       "   ['occupatio', 'exercendi']]),\n",
       " ('albertanus/albertanus.liberconsol.txt',\n",
       "  [['infremuit', 'spiritu'],\n",
       "   ['consolationem', 'inducit'],\n",
       "   ['uindicta', 'facienda'],\n",
       "   ['sanitatem', 'perducemus'],\n",
       "   ['probantur', 'incauta'],\n",
       "   ['deliberare', 'uolumus'],\n",
       "   ['iudicandum', 'properans'],\n",
       "   ['deprehensa', 'iudicare'],\n",
       "   ['insultare', 'coeperunt'],\n",
       "   ['propter', 'bonitatem'],\n",
       "   ['manifestare', 'mulieribus'],\n",
       "   ['consilium', 'habemus'],\n",
       "   ['pessimae', 'consulunt'],\n",
       "   ['consilium', 'liberauit'],\n",
       "   ['utrarumque', 'discretio'],\n",
       "   ['praestabit', 'gratiam'],\n",
       "   ['perseueranti', 'studio'],\n",
       "   ['nostram', 'doctrinam'],\n",
       "   ['imprudentiam', 'despicias'],\n",
       "   ['aliquatenus', 'erubescat'],\n",
       "   ['quibus', 'postulandum'],\n",
       "   ['mutandum', 'consilium'],\n",
       "   ['hominibus', 'exhibetur'],\n",
       "   ['ferramenta', 'commodaui'],\n",
       "   ['potiendum', 'incitantur'],\n",
       "   ['illecebris', 'excitari'],\n",
       "   ['debes', 'cupiditatem'],\n",
       "   ['conditionem', 'facere'],\n",
       "   ['consiliariis', 'ostendas'],\n",
       "   ['petentis', 'respiciunt'],\n",
       "   ['doctrinam', 'eloquii'],\n",
       "   ['nascuntur', 'innumerabilia'],\n",
       "   ['sermonibus', 'loquitur'],\n",
       "   ['prudentiae', 'innitaris'],\n",
       "   ['satiabitur', 'sanguine'],\n",
       "   ['reuerentiam', 'ostendunt'],\n",
       "   ['tantum', 'reuerentiam'],\n",
       "   ['philosophus', 'dixit'],\n",
       "   ['consiliariis', 'penitus'],\n",
       "   ['rerum', 'diligentissime'],\n",
       "   ['periculum', 'uersatur'],\n",
       "   ['uitiorum', 'prohibere'],\n",
       "   ['utilitas', 'assumenda'],\n",
       "   ['ipso', 'considerandum'],\n",
       "   ['circumuentos', 'esse'],\n",
       "   ['significationem', 'continet'],\n",
       "   ['dispositionem', 'permutes'],\n",
       "   ['aliquando', 'tristem'],\n",
       "   ['consilium', 'cepisti'],\n",
       "   ['congregare', 'paucos'],\n",
       "   ['multitudinis', 'stultorum'],\n",
       "   ['praeponant', 'amicitiae'],\n",
       "   ['concordiam', 'discordiae'],\n",
       "   ['uerbis', 'ostenderent'],\n",
       "   ['quales', 'consentiant'],\n",
       "   ['inculpatae', 'tutelae'],\n",
       "   ['posse', 'commoditatis'],\n",
       "   ['uindictam', 'faciendo'],\n",
       "   ['casibus', 'implicatur'],\n",
       "   ['scalis', 'ascendentibus'],\n",
       "   ['deberent', 'committi'],\n",
       "   ['puniendo', 'uindictam'],\n",
       "   ['timere', 'potestatem'],\n",
       "   ['imperium', 'habentem'],\n",
       "   ['fortunae', 'permittunt'],\n",
       "   ['prodesse', 'stultitia'],\n",
       "   ['iniuriam', 'relinquens'],\n",
       "   ['potestates', 'potius'],\n",
       "   ['praedictas', 'auctoritates'],\n",
       "   ['aequanimiter', 'portans'],\n",
       "   ['uindictam', 'faceres'],\n",
       "   ['moderamine', 'inculpatae'],\n",
       "   ['ratione', 'personarum'],\n",
       "   ['sustinere', 'cogimur'],\n",
       "   ['Magna', 'necessitate'],\n",
       "   ['necessitate', 'cogitur'],\n",
       "   ['possidentium', 'illas'],\n",
       "   ['uersatur', 'periculum'],\n",
       "   ['consuetudo', 'imitanda'],\n",
       "   ['grauioribus', 'autem'],\n",
       "   ['periculis', 'destructionem'],\n",
       "   ['turpitudini', 'anteponenda'],\n",
       "   ['uiolentiae', 'occurrere'],\n",
       "   ['resistere', 'iniuriae'],\n",
       "   ['consiliis', 'acquiescere'],\n",
       "   ['potestatem', 'concedat'],\n",
       "   ['responderunt', 'dicentes'],\n",
       "   ['facere', 'proposueras'],\n",
       "   ['facienda', 'laudauerunt'],\n",
       "   ['congregationem', 'magnam'],\n",
       "   ['fecistis', 'intrando'],\n",
       "   ['committere', 'uoluntati'],\n",
       "   ['oculorum', 'laetificat'],\n",
       "   ['philosophus', 'dixit'],\n",
       "   ['iniquissimum', 'mihi'],\n",
       "   ['adimplere', 'nollent'],\n",
       "   ['contumeliis', 'contra'],\n",
       "   ['imperium', 'custodiunt'],\n",
       "   ['siquidem', 'principum'],\n",
       "   ['nostram', 'mitigauerunt'],\n",
       "   ['nostram', 'suscipimus']]),\n",
       " ('albertanus/albertanus.sermo.txt',\n",
       "  [['Christi', 'misericordia'],\n",
       "   ['quidam', 'philosophus'],\n",
       "   ['poterit', 'comparari'],\n",
       "   ['dilectio', 'sapientiae'],\n",
       "   ['suscipit', 'incrementum'],\n",
       "   ['sapientiae', 'cupidus'],\n",
       "   ['dulcedinem', 'duobus'],\n",
       "   ['responsiones', 'habere'],\n",
       "   ['quidam', 'philosophus'],\n",
       "   ['penitus', 'repellere'],\n",
       "   ['clientelam', 'potentissimi'],\n",
       "   ['acquirere', 'debemus'],\n",
       "   ['confluentium', 'reducere'],\n",
       "   ['definiuit', 'iustitiam'],\n",
       "   ['causarum', 'gloriosae'],\n",
       "   ['affectare', 'debemus'],\n",
       "   ['litterarum', 'gloriosa'],\n",
       "   ['sepulchrum', 'patens']]),\n",
       " ('albertanus/albertanus1.txt',\n",
       "  [['aliquatenus', 'explicari'],\n",
       "   ['moribus', 'conformare'],\n",
       "   ['increpationes', 'insipiens'],\n",
       "   ['psalterium', 'suauem'],\n",
       "   ['epistolarum', 'dixit'],\n",
       "   ['maliuolentiam', 'quando'],\n",
       "   ['philosophus', 'dixit'],\n",
       "   ['quidam', 'philosophus'],\n",
       "   ['sermonibus', 'inuenitur'],\n",
       "   ['fraternitatis', 'caritatem'],\n",
       "   ['dereliquisti', 'querentes'],\n",
       "   ['exceptis', 'paruulis'],\n",
       "   ['habuerit', 'substantiam'],\n",
       "   ['clauserit', 'uiscera'],\n",
       "   ['diuitibus', 'subueniri'],\n",
       "   ['ablata', 'restituant'],\n",
       "   ['liberalitatem', 'tuam'],\n",
       "   ['paucitatem', 'ingenii']]),\n",
       " ('albertanus/albertanus2.txt',\n",
       "  [['longam', 'consuetudinem'],\n",
       "   ['homines', 'coagulatur'],\n",
       "   ['alterius', 'amicitiam'],\n",
       "   ['longam', 'consuetudinem'],\n",
       "   ['restringere', 'debes'],\n",
       "   ['uideant', 'uanitatem'],\n",
       "   ['quia', 'cogitationes'],\n",
       "   ['assiduitatem', 'quasi'],\n",
       "   ['alicuius', 'attendit'],\n",
       "   ['colloquium', 'stulti'],\n",
       "   ['insipiente', 'fortunato'],\n",
       "   ['clientelam', 'potentissimi'],\n",
       "   ['alius', 'philosophus'],\n",
       "   ['uindictam', 'querens'],\n",
       "   ['iracundus', 'prouocat'],\n",
       "   ['corrigitur', 'animus'],\n",
       "   ['consilium', 'ridentium'],\n",
       "   ['penitus', 'euitandam'],\n",
       "   ['mores', 'corrumpuntur'],\n",
       "   ['hominibus', 'laudaris'],\n",
       "   ['hominum', 'societatem'],\n",
       "   ['inexpugnabile', 'munimentum'],\n",
       "   ['obsistere', 'possunt'],\n",
       "   ['prouerbio', 'dicitur'],\n",
       "   ['iniquissimum', 'consilium'],\n",
       "   ['consilium', 'sequitur'],\n",
       "   ['adulantium', 'blanditiam'],\n",
       "   ['manifestare', 'debes'],\n",
       "   ['homines', 'expetenda'],\n",
       "   ['maledicenti', 'cuidam'],\n",
       "   ['refrigerabit', 'matrem'],\n",
       "   ['eradicat', 'fundamentum'],\n",
       "   ['iuuentute', 'peccarent'],\n",
       "   ['impudentiorem', 'intemperantia'],\n",
       "   ['plurimum', 'adiuuent'],\n",
       "   ['insistit', 'assiduus'],\n",
       "   ['aliquando', 'iuuenibus'],\n",
       "   ['sensato', 'habitabitur'],\n",
       "   ['studiosissime', 'laboraui'],\n",
       "   ['potestatem', 'habere'],\n",
       "   ['inopem', 'derelinquas'],\n",
       "   ['lenitate', 'tractandi'],\n",
       "   ['epistolarum', 'dixit'],\n",
       "   ['propter', 'uerecundiam'],\n",
       "   ['promissiones', 'fiunt'],\n",
       "   ['multam', 'instantiam'],\n",
       "   ['occupare', 'antequam'],\n",
       "   ['dissoluit', 'amicitiam'],\n",
       "   ['temere', 'manifestat'],\n",
       "   ['cotidie', 'recordetur'],\n",
       "   ['familiarem', 'loquelam'],\n",
       "   ['quicquid', 'desiderasti'],\n",
       "   ['contumeliam', 'donis'],\n",
       "   ['quidam', 'philosophus'],\n",
       "   ['collatum', 'commemorare'],\n",
       "   ['dixit', 'philosophus'],\n",
       "   ['rationabili', 'causa'],\n",
       "   ['assiduam', 'conuersationem']]),\n",
       " ('albertanus/albertanus3.txt',\n",
       "  [['dicuntur', 'corporales'],\n",
       "   ['temporalium', 'tractabimus'],\n",
       "   ['euangelio', 'legitur'],\n",
       "   ['philosophus', 'dixit'],\n",
       "   ['Philippenses', 'dixit'],\n",
       "   ['posterior', 'iudicat'],\n",
       "   ['suspicantur', 'qualia'],\n",
       "   ['facile', 'suspicatur'],\n",
       "   ['quidam', 'philosophus'],\n",
       "   ['conscientiam', 'tuam'],\n",
       "   ['philosophus', 'dixit'],\n",
       "   ['saturabitur', 'panibus'],\n",
       "   ['inculta', 'relinquant'],\n",
       "   ['lapidum', 'destructa'],\n",
       "   ['didici', 'disciplinam'],\n",
       "   ['reprehendendi', 'sunt'],\n",
       "   ['patrimonia', 'effudere'],\n",
       "   ['tanta', 'assequuntur'],\n",
       "   ['largitionem', 'fundum'],\n",
       "   ['ipsis', 'mediocritatis'],\n",
       "   ['inducant', 'debilitatem'],\n",
       "   ['cupiditati', 'detrahendum'],\n",
       "   ['uocabatur', 'Rupilius'],\n",
       "   ['homine', 'possideantur'],\n",
       "   ['homine', 'possideantur'],\n",
       "   ['facias', 'paupertati'],\n",
       "   ['periculis', 'consuetudo'],\n",
       "   ['ipso', 'considerandum'],\n",
       "   ['resistere', 'iniurie'],\n",
       "   ['moderamine', 'inculpate'],\n",
       "   ['quamque', 'consequitur'],\n",
       "   ['corporis', 'facienda'],\n",
       "   ['hominibus', 'amandus'],\n",
       "   ['habentis', 'potestatem'],\n",
       "   ['testamenti', 'probari'],\n",
       "   ['iudicare', 'potestis'],\n",
       "   ['ecclesiam', 'competentem'],\n",
       "   ['necessitas', 'subsit'],\n",
       "   ['fugam', 'conuerteris'],\n",
       "   ['distuleris', 'persequi'],\n",
       "   ['nobiles', 'thesauros'],\n",
       "   ['administrari', 'ciuitas'],\n",
       "   ['ingenii', 'utilitatem']]),\n",
       " ('albertanus/albertanus4.txt',\n",
       "  [['negotiis', 'implicantur'],\n",
       "   ['iniustitiam', 'omnia'],\n",
       "   ['gentem', 'transfertur'],\n",
       "   ['gentem', 'transfertur'],\n",
       "   ['amplectenda', 'negotiis'],\n",
       "   ['implicatur', 'erroribus'],\n",
       "   ['epistolarum', 'dixit'],\n",
       "   ['relinquunt', 'hominem'],\n",
       "   ['uitiorum', 'exemplis'],\n",
       "   ['uirtutes', 'accedendum'],\n",
       "   ['Timotheum', 'secundam'],\n",
       "   ['hominibus', 'superbia'],\n",
       "   ['speratur', 'potentior'],\n",
       "   ['continuationem', 'studii'],\n",
       "   ['exercitationes', 'ingenii'],\n",
       "   ['philosophus', 'dixit'],\n",
       "   ['causa', 'benignitatis'],\n",
       "   ['beniuolis', 'officium'],\n",
       "   ['lacrimosos', 'faciunt'],\n",
       "   ['rationis', 'reuocare'],\n",
       "   ['periculorum', 'susceptio'],\n",
       "   ['tribuendo', 'consistit'],\n",
       "   ['initium', 'spectandum'],\n",
       "   ['circumspicienda', 'sunt'],\n",
       "   ['timet', 'uerecundiam'],\n",
       "   ['spiritu', 'ambulando'],\n",
       "   ['frequens', 'cogitatio'],\n",
       "   ['philosophus', 'dixit'],\n",
       "   ['excogitari', 'possunt'],\n",
       "   ['conposita', 'uideatur'],\n",
       "   ['argumentum', 'conposite'],\n",
       "   ['funiculus', 'triplex'],\n",
       "   ['caritatem', 'amplitudinem'],\n",
       "   ['tranquillitatem', 'expetentes'],\n",
       "   ['opusculum', 'emendabis']])]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "latin_library_reader = get_corpus_reader(corpus_name='latin_text_latin_library', language='latin')\n",
    "latin_split_words = get_split_words(latin_library_reader, latin_trie)\n",
    "print(f'{len(latin_split_words)} files with oddly joined words out of {len(latin_library_reader.fileids())} corpus files')\n",
    "list(latin_split_words.items())[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 293/293 [03:21<00:00,  5.72files/s]     | 1/293 [00:05<26:14,  5.39s/files]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "112 files with oddly joined words out of 293 corpus files\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('ammianus-marcellinus__rerum-gestarum__latin.json',\n",
       "  [['constr', 'ingerentur'],\n",
       "   ['inter', 'clamantibus'],\n",
       "   ['incorruptis', 'simum']]),\n",
       " ('apuleius__apologia__latin.json',\n",
       "  [['paupertatem', 'philosopho'], ['ostendis', 'humanissimo']]),\n",
       " ('ausonius-decimus-magnus__caesares__latin.json',\n",
       "  [['sequentes', 'expediam']]),\n",
       " ('ausonius-decimus-magnus__commemoratio-professorum-burdigalensium__latin.json',\n",
       "  [['etenim', 'commemorare'],\n",
       "   ['cathedrae', 'perdidit'],\n",
       "   ['noster', 'commemorauit'],\n",
       "   ['tamen', 'grammatices'],\n",
       "   ['dignus', 'grammaticos'],\n",
       "   ['commemoratus', 'Urbice'],\n",
       "   ['magistrum', 'collegam'],\n",
       "   ['nuncupant', 'Apollinares'],\n",
       "   ['carminum', 'orationem'],\n",
       "   ['genitori', 'conlatus'],\n",
       "   ['solstitialis', 'uelut'],\n",
       "   ['disciplinis', 'adpulit']]),\n",
       " ('ausonius-decimus-magnus__eclogarum-liber__latin.json',\n",
       "  [['Prometheus', 'testatur'],\n",
       "   ['curis', 'sollicitudo'],\n",
       "   ['addens', 'quadrantem'],\n",
       "   ['quattuor', 'feruidis'],\n",
       "   ['uoltu', 'perstrictus'],\n",
       "   ['quadrigis', 'iungunt'],\n",
       "   ['commeditantes', 'murmure'],\n",
       "   ['uacillat', 'conuulsae'],\n",
       "   ['cohortis', 'septimus']]),\n",
       " ('ausonius-decimus-magnus__ephemeris__latin.json',\n",
       "  [['gehennae', 'anticipat'], ['saeuum', 'insidiatorem']]),\n",
       " ('ausonius-decimus-magnus__epicedion-in-patrem__latin.json',\n",
       "  [['inemptae', 'officium']]),\n",
       " ('ausonius-decimus-magnus__epigrammaton-liber__latin.json',\n",
       "  [['secundus', 'Danuuius'],\n",
       "   ['subisti', 'supplicium'],\n",
       "   ['comparatis', 'similem'],\n",
       "   ['parasset', 'exteriore'],\n",
       "   ['haruspicium', 'tractauit'],\n",
       "   ['papillas', 'Hermiones']]),\n",
       " ('ausonius-decimus-magnus__epistulae__latin.json',\n",
       "  [['absentiae', 'necessitatem'],\n",
       "   ['temperans', 'torpere'],\n",
       "   ['uagantes', 'persequeris'],\n",
       "   ['recuses', 'stringere'],\n",
       "   ['lacertos', 'caesariem'],\n",
       "   ['nitentes', 'descendis'],\n",
       "   ['Mosellae', 'protulerat'],\n",
       "   ['fouebat', 'temperies'],\n",
       "   ['uolentum', 'accipiat'],\n",
       "   ['habenis', 'fulmineum'],\n",
       "   ['laborum', 'destituor'],\n",
       "   ['triumphis', 'grandia'],\n",
       "   ['minister', 'credidit'],\n",
       "   ['renatum', 'Pythagoram'],\n",
       "   ['paternis', 'claudente'],\n",
       "   ['Bellerophontis', 'mens'],\n",
       "   ['dulces', 'fermentare'],\n",
       "   ['creditoribus', 'bonus'],\n",
       "   ['pecuniam', 'restituet'],\n",
       "   ['prosint', 'supplicibus'],\n",
       "   ['senilem', 'sollicitent'],\n",
       "   ['carentia', 'Hellespontiaci']]),\n",
       " ('ausonius-decimus-magnus__epitaphia__latin.json',\n",
       "  [['tumulos', 'anguineis'], ['sepulcrum', 'constituit']])]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perseus_latin_reader = get_corpus_reader(corpus_name='latin_text_perseus', language='latin')\n",
    "perseus_split_words = get_split_words(perseus_latin_reader, latin_trie) \n",
    "print(f'{len(perseus_split_words)} files with oddly joined words out of {len(perseus_latin_reader.fileids())} corpus files')\n",
    "list(perseus_split_words.items())[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "### The word splitting is effective and the data generally looks convincing; however it is most useful when paired with some supervision. There are a few edge cases, such sometimes prepositions are split away needlessly from compound verbs (compounding is a regular linguistic trend and variety is expected). Depending on your requirements, needless splits may be acceptable, for example, if you're looking to build a high quality embedding, splitting may help cluster meanings in a word vector representation. Auto-splitting is a tool for your toolbox and a suggestion to be considered, depending on your needs and use of the corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving & Restoring the Word Trie for later use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('latin.word_trie.pkl', 'wb') as writer:\n",
    "    pickle.dump(latin_trie, writer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_new_trie = None\n",
    "with open('latin.word_trie.pkl', 'rb') as reader:\n",
    "    my_new_trie = pickle.load(reader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# prove that the reconstituted trie can be used:\n",
    "my_new_trie.has_word('et')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This word_trie will be used in other notebooks, but for now\n",
    "## That's all for now for now folks! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}