lm3-portuguese.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "e38c0aad-6ca2-456d-b908-cbbec3c9e34d"
    }
   },
   "source": [
    "# (MultiFiT) Portuguese Bidirectional Language Model (LM) from scratch\n",
    "### (architecture 4 QRNN with 1550 hidden parameters by layer, SentencePiece tokenizer and hyperparameters from the MultiFiT method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n",
    "- Date: September 2019\n",
    "- Post in medium: [link](https://medium.com/@pierre_guillou/nlp-fastai-portuguese-language-model-980c8ec75362)\n",
    "- Ref: [Fastai v1](https://docs.fast.ai/) (Deep Learning library on PyTorch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Information**\n",
    "\n",
    "According to this new article \"[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)\" (September 10, 2019), the architecture QRNN and the SentencePiece tokenizer give better results than AWD-LSTM and the spaCy tokenizer respectively. Therefore, they have been used in this notebook to train a Portuguese Bidirectional Language Model on a Wikipedia corpus of 100 millions tokens. \n",
    "\n",
    "More, the hyperparameters values given at the end of the article have been used, too.\n",
    "\n",
    "**Wikipedia corpus**\n",
    "- download: 193 651 articles of 182 536 221 tokens\n",
    "- used: 166 580 articles of 100 255 322 tokens\n",
    "\n",
    "**Hyperparameters values**\n",
    "- (batch size) bs = 50\n",
    "- (QRNN) 4 QRNN (default: 3) with 1550 hidden parameters each one (default: 1152)\n",
    "- (SentencePiece) vocab of 15000 tokens\n",
    "- (dropout) mult_drop = 0\n",
    "- (weight decay) wd = 0.01\n",
    "- (number of training epochs) 10 epochs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Portuguese Bidirectional Language Model with the [MultiFiT](https://arxiv.org/abs/1909.04761) configuration performs well in both directions (see following results).\n",
    "\n",
    "**To be noticed**: we could have trained on more epochs both the forward and backward models but we decided to follow the MultiFiT training method based on the tests done by Jeremy Howard and his fastai team that shows that a Text Classifier fine-tuned with a specialized LM itself fine-tuned with a general LM performs better if the general LM was not trained too much. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- forward : (accuracy) 39.68% | (perplexity) 21.76\n",
    "- backward: (accuracy) 43.67% | (perplexity) 22.16"
   ]
  },
      {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**To be improved**"
   ]
  },
    {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lost function FlattenedLoss of LabelSmoothingCrossEntropy should be tested as it is used in the MultiFiT method (see the notebook [lm3-portuguese-classifier-TCU-jurisprudencia.ipynb](https://github.com/piegu/language-models/blob/master/lm3-portuguese-classifier-TCU-jurisprudencia.ipynb) to get the code)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Initialisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "151cd18f-76e3-440f-a8c7-ffa5c6b5da01"
    }
   },
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "96f02439-3586-4c9d-8c34-aa7c3b17a0a6"
    }
   },
   "outputs": [],
   "source": [
    "# batch size to be choosen according to your GPU \n",
    "# bs=48\n",
    "# bs=24\n",
    "bs=50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6ceb4db2-e4cf-4fe0-a393-91df4a7ed3e7"
    }
   },
   "outputs": [],
   "source": [
    "torch.cuda.set_device(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6329e650-fc03-4323-ac0c-3aa280e0de91"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fastai: 1.0.57\n",
      "cuda: True\n"
     ]
    }
   ],
   "source": [
    "import fastai\n",
    "print(f'fastai: {fastai.__version__}')\n",
    "print(f'cuda: {torch.cuda.is_available()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6f24e68b-3df0-4997-8a50-3a37ea6a5257"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\r\n",
      "```text\r\n",
      "=== Software === \r\n",
      "python        : 3.7.4\r\n",
      "fastai        : 1.0.57\r\n",
      "fastprogress  : 0.1.21\r\n",
      "torch         : 1.2.0\r\n",
      "nvidia driver : 410.104\r\n",
      "torch cuda    : 10.0.130 / is available\r\n",
      "torch cudnn   : 7602 / is enabled\r\n",
      "\r\n",
      "=== Hardware === \r\n",
      "nvidia gpus   : 1\r\n",
      "torch devices : 1\r\n",
      "  - gpu0      : 16130MB | Tesla V100-SXM2-16GB\r\n",
      "\r\n",
      "=== Environment === \r\n",
      "platform      : Linux-4.9.0-9-amd64-x86_64-with-debian-9.9\r\n",
      "distro        : #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11)\r\n",
      "conda env     : base\r\n",
      "python        : /opt/anaconda3/bin/python\r\n",
      "sys.path      : /home/jupyter/tutorials/fastai/course-nlp\r\n",
      "/opt/anaconda3/lib/python37.zip\r\n",
      "/opt/anaconda3/lib/python3.7\r\n",
      "/opt/anaconda3/lib/python3.7/lib-dynload\r\n",
      "/opt/anaconda3/lib/python3.7/site-packages\r\n",
      "/opt/anaconda3/lib/python3.7/site-packages/IPython/extensions\r\n",
      "```\r\n",
      "\r\n",
      "Please make sure to include opening/closing ``` when you paste into forums/github to make the reports appear formatted as code sections.\r\n",
      "\r\n",
      "Optional package(s) to enhance the diagnostics can be installed with:\r\n",
      "pip install distro\r\n",
      "Once installed, re-run this utility to get the additional information\r\n"
     ]
    }
   ],
   "source": [
    "!python -m fastai.utils.show_install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "194a6989-31f1-4702-b94d-32b974ded8e6"
    }
   },
   "outputs": [],
   "source": [
    "data_path = Config.data_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "cf070ab7-babb-4cf0-a315-401f65461dc8"
    }
   },
   "source": [
    "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents. (For other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "70da588b-8af1-4f97-97c2-c9f2d4d46e1a"
    }
   },
   "outputs": [],
   "source": [
    "lang = 'pt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "701ab344-0430-4f43-bbe2-337a12cae6be"
    }
   },
   "outputs": [],
   "source": [
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)\n",
    "\n",
    "lm_fns3 = [f'{lang}_wt_sp15_multifit', f'{lang}_wt_vocab_sp15_multifit']\n",
    "lm_fns3_bwd = [f'{lang}_wt_sp15_multifit_bwd', f'{lang}_wt_vocab_sp15_multifit_bwd']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "nbpresent": {
     "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2"
    }
   },
   "source": [
    "## Data (Portuguese wikipedia)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "nbpresent": {
     "id": "4e67d876-c7d0-4bdf-a6f9-ae06ae1fc023"
    }
   },
   "source": [
    "### Download data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "dd2fd658-b690-484c-b60a-69dc6b7bf384"
    }
   },
   "outputs": [],
   "source": [
    "from nlputils import split_wiki,get_wiki\n",
    "from nlputils2 import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "28c01920-f13c-493e-9a97-e5b2c24133a8"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jupyter/.fastai/data/ptwiki/ptwiki already exists; not downloading\n",
      "CPU times: user 0 ns, sys: 0 ns, total: 0 ns\n",
      "Wall time: 273 µs\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "get_wiki(path,lang)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This function splits the single wikipedia file into a separate file per article. This is often easier to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "100000\n",
      "200000\n",
      "300000\n",
      "400000\n",
      "500000\n",
      "600000\n",
      "700000\n",
      "800000\n",
      "900000\n",
      "1000000\n",
      "1100000\n",
      "1200000\n",
      "1300000\n",
      "1400000\n",
      "1500000\n",
      "1600000\n",
      "1700000\n",
      "1800000\n",
      "1900000\n",
      "2000000\n",
      "2100000\n",
      "2200000\n",
      "2300000\n",
      "2400000\n",
      "2500000\n",
      "2600000\n",
      "2700000\n",
      "2800000\n",
      "2900000\n",
      "3000000\n",
      "3100000\n",
      "3200000\n",
      "3300000\n",
      "3400000\n",
      "3500000\n",
      "3600000\n",
      "3700000\n",
      "3800000\n",
      "3900000\n",
      "4000000\n",
      "4100000\n",
      "4200000\n",
      "4300000\n",
      "4400000\n",
      "4500000\n",
      "4600000\n",
      "4700000\n",
      "4800000\n",
      "4900000\n",
      "5000000\n",
      "5100000\n",
      "5200000\n",
      "5300000\n",
      "5400000\n",
      "5500000\n",
      "5600000\n",
      "5700000\n",
      "5800000\n",
      "5900000\n",
      "6000000\n",
      "6100000\n",
      "6200000\n",
      "6300000\n",
      "6400000\n",
      "6500000\n",
      "6600000\n",
      "6700000\n",
      "6800000\n",
      "6900000\n",
      "7000000\n",
      "CPU times: user 14 s, sys: 5.58 s, total: 19.5 s\n",
      "Wall time: 21.4 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/jupyter/.fastai/data/ptwiki/docs')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "split_wiki2(path,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e6eae780-775e-45e9-9b99-b8a87d5fb8ff"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/ptwiki/models'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/corpus_100000000_old'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/pt_databunch_corpus_100'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/pt_databunch_corpus_100_bwd'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/docs'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/ptwiki-latest-pages-articles.xml.bz2'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/docs_old'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/wikiextractor'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/ptwiki'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/ptwiki-latest-pages-articles.xml'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/log')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e1ac63e7-1cbb-4996-838d-dc58446a65ef"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<doc id=\"220\" url=\"https://pt.wikipedia.org/wiki?curid=220\" title=\"Astronomia\">\r\n",
      "Astronomia\r\n",
      "\r\n",
      "Astronomia é uma ciência natural que estuda corpos celestes (como estrelas, planetas, cometas, nebulosas, aglomerados de estrelas, galáxias) e fenômenos que se originam fora da atmosfera da Terra (como a radiação cósmica de fundo em micro-ondas). Preocupada com a evolução, a física, a química e o movimento de objetos celestes, bem como a formação e o desenvolvimento do universo.\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {path}/{name}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "d23e0ef7-21e5-4cc5-945d-60ee33c02ce3"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "folder = \"docs\"\n",
    "clean_files(path,folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "92b0b087-a6a8-403a-a7a1-d9b47757e5cf"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/ptwiki/docs/Pac_Man__personagem_.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/docs/Jo_o_Comneno_Vatatzes.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/docs/Lupo_Servato.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/docs/Maom_.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/ptwiki/docs/Ilha_de_S_o_Sim_o__Galiza_.txt')]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dest = path/'docs'\n",
    "dest.ls()[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pac-Man é o personagem principal da franquia de jogos eletrônicos criada pela Namco, introduzido pelo jogo de arcade Pac-Man de 1980. É marcado como um dos personagens mais icônicos dos games desde sua estréia, recebendo inúmeros jogos inspirados no mesmo e programas de TV. O personagem também serve como mascote oficial da empresa Namco.\r\n",
      "\r\n",
      "Pac-Man a princípio era um círculo amarelo com uma boca que constantemente abria e fechava. Somente a partir da primeira série animada lançada em 1982 é que o personagem ganhou uma aparência humanoide tendo braços e pernas com luvas e sapatos, além de olhos e um nariz, e também um chapéu marrom. Essa mesma aparência serviu de imagem promocional para os demais jogos lançados na época, porém sem o chapéu do desenho. Com o tempo seus olhos foram se alterando e passaram a ser duas bolinhas pretas com uma abertura nos lados típico de desenhos clássicos.\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {dest.ls()[0]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "nbpresent": {
     "id": "575fa672-7b3a-4238-923f-ec929d3a00ee"
    }
   },
   "source": [
    "### Size of downloaded data in the docs folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "270470c2-e0eb-446a-9654-de6c45bc4f0d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "193651 files - 182536221 tokens\n",
      "CPU times: user 17.1 s, sys: 3.56 s, total: 20.6 s\n",
      "Wall time: 1min 17s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "num_files, num_tokens = get_num_tokens(dest)\n",
    "print(f'{num_files} files - {num_tokens} tokens')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "nbpresent": {
     "id": "daae36a0-d90b-45ad-b0e3-a2cd56ce7079"
    }
   },
   "source": [
    "### Create a corpus of about 100 millions of tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6c383e0e-f4f6-46e5-9f54-469437d66f07"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "files copied to the new corpus folder: /home/jupyter/.fastai/data/ptwiki/corpus_100000000\n",
      "CPU times: user 6.88 s, sys: 5.79 s, total: 12.7 s\n",
      "Wall time: 19.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "path_corpus = get_corpus(dest, path, num_tokens, obj_tokens=1e8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "b597308e-9521-4637-9e84-85a6cd2ace85"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "166580 files - 100255322 tokens\n",
      "CPU times: user 10.9 s, sys: 2.16 s, total: 13.1 s\n",
      "Wall time: 55.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# VERIFICATION of the number of words in the corpus folder\n",
    "num_files_corpus, num_tokens_corpus = get_num_tokens(path_corpus)\n",
    "print(f'{num_files_corpus} files - {num_tokens_corpus} tokens')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# change name of the corpus \n",
    "!mv {path}/'corpus_100000000' {path}/'corpus2_100'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "dest = path/'corpus2_100'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 39min 42s, sys: 24.9 s, total: 40min 7s\n",
      "Wall time: 20min 17s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = (TextList.from_folder(dest, processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "        .split_by_rand_pct(0.1, seed=42)\n",
    "        .label_for_lm()\n",
    "        .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data.save(f'{path}/{lang}_databunch_corpus2_100_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15000, 149922)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.vocab.itos),len(data.train_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15000, 15000)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.vocab.itos),len(data.vocab.stoi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text ▁xxbos ▁xxmaj ▁joão ▁xxmaj ▁com ne no ▁xxmaj ▁ va ta tz es ▁( ), ▁ou ▁simplesmente ▁xxmaj ▁joão ▁xxmaj ▁com ne no ▁ou ▁xxmaj ▁joão ▁xxmaj ▁ va ta tz es , ▁foi ▁um ▁importante ▁político ▁e ▁líder ▁militar ▁bizantino ▁durante ▁os ▁reinado s ▁de ▁e ▁ . ▁xxmaj ▁ele ▁nasceu ▁por ▁volta ▁de ▁e ▁morreu ▁de ▁causas ▁naturais ▁durante ▁uma ▁revolta ▁que ▁ele ▁mesmo ▁iniciou ▁contra ▁em ▁ . ▁xxmaj ▁joão ▁xxmaj ▁com ne no ▁xxmaj ▁ va ta tz es ▁era ▁filho ▁do ▁se bas to hi per ta to ▁xxmaj ▁teodoro ▁xxmaj ▁ va ta tz es ▁e ▁da ▁por fi ro gê nita ▁xxmaj ▁eu dó xia ▁xxmaj ▁com ne na , ▁princesa ▁filha ▁do ▁imperador ▁e ▁xxmaj ▁irene ▁da ▁xxmaj ▁hungria . ▁xxmaj ▁teodoro ▁era ▁um ▁dos ▁\" hom ens - novo s \" ▁alça dos ▁por ▁xxmaj ▁joão ▁xxup ▁ii ; ▁a ▁família ▁xxmaj ▁ va ta tz es ▁não ▁estava ▁entre ▁as ▁mais ▁proeminente s ▁da ▁aristocracia ▁bizantina , ▁mesmo ▁sendo ▁muito ▁importante ▁nas ▁cerca nia s ▁de ▁xxmaj ▁adrian ópolis , ▁na ▁xxmaj ▁ tr ácia . ▁xxmaj ▁os ▁pais ▁de ▁xxmaj ▁joão ▁se ▁casa ram ▁em ▁11 31 ▁e ▁ele ▁nasceu ▁logo ▁em ▁seguida , ▁provavelmente ▁por ▁volta ▁de ▁11 32 . ▁xxmaj ▁joão ▁tinha ▁um ▁irmão , ▁xxmaj ▁and r ônico , ▁que ▁também ▁era ▁um ▁importante ▁general ▁ - ▁ele ▁liderou ▁um ▁exército ▁contra ▁a ▁cidade ▁de ▁xxmaj ▁ama se ia ▁em ▁11 76 ▁e ▁terminou ▁morto ▁pelos ▁turco s ▁se l jú cida s ; ▁eles ▁mostra ram ▁a ▁cabeça ▁dele ▁na ▁xxmaj ▁batalha ▁de ▁xxmaj ▁mi rio cé fa lo ▁que ▁se ▁seguiu . ▁xxmaj ▁ele ▁teve ▁também ▁outro ▁irmão , ▁chamado ▁xxmaj ▁aleixo . ▁a ▁esposa ▁de ▁xxmaj ▁joão ▁era ▁chamada ▁xxmaj ▁maria ▁xxmaj ▁du cena ▁e ▁eles ▁tiveram ▁dois ▁filhos , ▁xxmaj ▁aleixo ▁e ▁xxmaj ▁manuel , ▁o ▁último ▁batizado ▁em ▁homenagem ▁ao ▁tio ▁de ▁xxmaj ▁joão , ▁o ▁imperador ▁xxmaj ▁manuel , ▁a ▁quem ▁xxmaj ▁joão ▁era ▁muito ▁de vo tado ▁ - ▁a ▁ponto ▁de ▁permitir ▁que ▁tivesse ▁um ▁caso ▁com ▁sua ▁irmã ▁xxmaj ▁teo dora . ▁xxmaj ▁joão ▁aparece ▁nas ▁fontes ▁da ▁época ▁como ▁um ▁general ▁graduado ▁na ▁década ▁de ▁11 70 ; ▁é ▁certo ▁que ▁ele ▁serviu ▁em ▁cargos ▁menores ▁antes ▁disso , ▁mas ▁não ▁há ▁registros ▁sobreviventes ▁sobre ▁esta ▁fase ▁de ▁sua ▁vida . ▁xxmaj ▁ele ▁indu bi ta velmente ▁foi ▁aprendiz ▁do ▁pai , ▁xxmaj ▁teodoro , ▁que ▁era ▁também ▁um ▁general , ▁responsável ▁pelo ▁cerco ▁de ▁xxmaj ▁ze mun o ▁na ▁fronteira ▁h ún gar a ▁em ▁11 51 ▁e ▁conquistado r ▁de ▁xxmaj ▁tar so ▁na ▁xxmaj ▁ cil ícia ▁em ▁11 58 . ▁xxmaj ▁em ▁11 76 , ▁o ▁imperador ▁xxmaj ▁manuel ▁tentou ▁destruir ▁o ▁xxmaj ▁sul ta nato ▁xxmaj ▁se l jú cida ▁de ▁xxmaj ▁ rum , ▁mas ▁foi ▁derrotado ▁em ▁xxmaj ▁mi rio cé fa lo . ▁xxmaj ▁após ▁uma ▁ t ré gua ▁que ▁permitiu ▁que ▁o ▁exército ▁bizantino ▁saí sse ▁do ▁território ▁turco , ▁xxmaj ▁manuel ▁não ▁cumpriu ▁com ▁todas ▁as ▁condições ▁acorda dos , ▁principalmente ▁a ▁destruição ▁das ▁fortaleza s ▁na ▁fronteira , ▁exigência ▁do ▁sultão ▁se l jú cida ▁xxmaj ▁qui li je ▁xxmaj ▁ar s lam ▁xxup ▁ii ▁como ▁pré - re qui s ito ▁para ▁o ▁fim ▁das ▁hostil idades . ▁a ▁fortaleza ▁de ▁foi ▁arras ada , ▁mas ▁xxmaj ▁do ri le ia , ▁muito ▁mais ▁importante , ▁não ▁foi . ▁a ▁reação ▁do ▁sultão ▁foi ▁enviar ▁uma ▁nada ▁de sp rez ível ▁força ▁de ▁cavalaria , ▁com ▁aproximadamente ▁homens , ▁para ▁pilha r ▁o ▁território ▁bizantino ▁no ▁vale ▁do ▁xxmaj ▁me andro , ▁na ▁xxmaj ▁anatólia ▁xxmaj ▁ocidental . ▁xxmaj ▁joão ▁xxmaj ▁com ne no ▁xxmaj ▁ va ta tz es ▁foi ▁encarregado ▁de ▁comandar ▁o ▁exército ▁bizantino ▁e ▁partiu ▁de ▁xxmaj ▁constantinopla ▁com ▁ordens ▁de ▁intercepta r ▁os ▁invasores . ▁xxmaj ▁seus ▁tenente s ▁eram ▁xxmaj ▁constantino ▁xxmaj ▁du cas ▁e ▁xxmaj ▁miguel ▁xxmaj ▁as pie ta ▁e ▁seu ▁exército ▁foi ▁sendo ▁reforça do ▁conforme ▁ iam ▁avança ndo ▁por ▁território ▁bizantino . ▁xxmaj ▁ va ta tz es ▁inter cept ou ▁o ▁exército ▁se l jú cida ▁quando ▁ele ▁retorna va ▁para ▁o ▁território ▁do ▁xxmaj ▁sul ta nato ▁carregado ▁com ▁os ▁ esp ólio s ▁dos ▁saque s ▁às ▁cidades ▁bizantina s . ▁xxmaj ▁ele ▁então ▁dis pô s ▁seu ▁exército ▁de ▁forma ▁a ▁criar ▁uma ▁em bos cada ▁clássica , ▁e ▁iniciou ▁seu ▁ataque ▁quanto ▁os ▁turco s ▁estavam ▁cruz ando ▁o ▁xxmaj ▁me andro , ▁perto ▁dos ▁assentamento s ▁de ▁xxmaj ▁hi élio ▁e ▁xxmaj ▁li mo qui r . ▁o ▁exército ▁se l jú cida ▁estava ▁quase ▁inde fe so ▁e ▁foi ▁completamente ▁destruído ; ▁o ▁historiador ▁bizantino ▁xxmaj ▁ nice tas ▁xxmaj ▁con ia tes ▁afirmou ▁que ▁apenas ▁uns ▁poucos ▁entre ▁os ▁milhares ▁de ▁turco s ▁sobreviveram . ▁o ▁comandante ▁se l jú cida , ▁que ▁tinha ▁o ▁título ▁de ▁a ta be gue , ▁foi ▁morto ▁quando ▁tentava ▁escapar ▁da ▁armadilha . ▁xxmaj ▁esta ▁batalha ▁foi ▁uma ▁importante ▁vitória ▁para ▁os ▁bizantino s ▁e ▁serviu ▁de ▁exemplo ▁de ▁ qu ão ▁limitado s ▁foram ▁os ▁efeitos ▁da ▁derrota ▁em ▁xxmaj ▁mi rio cé fa lo ▁sobre ▁o ▁controle ▁bizantino ▁sobre ▁a ▁xxmaj ▁anatólia . ▁a ▁vitória ▁foi ▁seguida ▁de ▁outros ▁ ra ide s ▁puni tivos ▁contra ▁os ▁ nô made s ▁turco s ▁da ▁região ▁do ▁vale ▁do ▁xxmaj ▁me andro . ▁xxmaj ▁ va ta tz es ▁aparece ▁novamente ▁nas ▁fontes ▁em ▁11 82 , ▁desta ▁vez ▁com ▁uma ▁posição ▁bastante ▁alta : ▁ele ▁era ▁grande ▁doméstico , ▁o ▁comandante - em - chefe ▁do ▁exército ▁bizantino , ▁e ▁também ▁o ▁ estra te go ▁do ▁importante ▁xxmaj ▁tema ▁da ▁xxmaj ▁ tr ácia . ▁a ▁cidade ▁de ▁xxmaj ▁adrian ópolis ▁era ▁tanto ▁a ▁capital ▁do ▁tema ▁como ▁também ▁a ▁base ▁do ▁poder ▁da ▁família ▁xxmaj ▁ va ta tz es ▁e ▁xxmaj ▁joão ▁aparece ▁como ▁construtor ▁e ▁patrocinador ▁de ▁hospitais ▁e ▁a si los ▁na ▁região . ▁xxmaj ▁após ▁a ▁morte ▁de ▁xxmaj ▁manuel ▁i ▁em ▁11 80 , ▁seu ▁filho ▁xxmaj ▁aleixo ▁xxup ▁ii ▁xxmaj ▁com ne no ▁assumiu ▁o ▁trono . ▁xxmaj ▁como ▁ele ▁era ▁apenas ▁uma ▁criança , ▁o ▁poder ▁estava ▁nas ▁mãos ▁de ▁sua ▁mãe , ▁a ▁imperatriz ▁xxmaj ▁maria ▁de ▁xxmaj ▁antioquia . ▁xxmaj ▁porém , ▁seu ▁governo ▁era ▁imp op ular , ▁principalmente ▁entre ▁a ▁aristocracia , ▁que ▁ res senti a ▁a ▁sua ▁origem ▁latina . ▁xxmaj ▁quando ▁o ▁primo ▁de ▁xxmaj ▁manuel , ▁xxmaj ▁and r ônico ▁xxmaj ▁com ne no ▁tentou ▁tomar ▁o ▁poder ▁no ▁início ▁de ▁11 82 , ▁ele ▁escreveu ▁para ▁xxmaj ▁ va ta tz es ▁numa ▁tentativa ▁de ▁sub or ná - lo . ▁xxmaj ▁joão ▁reconheceu ▁xxmaj ▁and r ônico ▁como ▁um ▁tira no ▁potencial ▁e ▁escreveu ▁de ▁volta ▁em ▁termos ▁insu l tuoso s . ▁o ▁primo ▁de ▁xxmaj ▁ va ta tz es , ▁xxmaj ▁and r ônico ▁xxmaj ▁contos te fa no , ▁o ▁comandante ▁da ▁marinha ▁bizantina , ▁porém , ▁foi ▁ ilu dido ▁e ▁teve ▁um ▁papel ▁pre pon der ante ▁no ▁golpe ▁ao ▁permitir ▁que ▁as ▁forças ▁de ▁xxmaj ▁and r ônico ▁in va di ssem ▁xxmaj ▁constantinopla . ▁xxmaj ▁uma ▁vez ▁no ▁poder , ▁xxmaj ▁and r ônico ▁xxmaj ▁com ne no ▁se ▁mostrou ▁de ▁fato ▁um ▁tira no ▁im bu ído ▁de ▁um ▁ar dente ▁desejo ▁de ▁rompe r ▁o ▁poder ▁e ▁influência ▁das ▁famílias ▁ar isto cráti cas ▁bizantina s . ▁xxmaj ▁no ▁final ▁de ▁11 82 , ▁com ▁xxmaj ▁and r ônico ▁firme ▁no ▁poder , ▁xxmaj ▁ va ta tz es ▁aparece ▁mora ndo ▁perto ▁de ▁xxmaj ▁fila dé l fia , ▁na ▁xxmaj ▁anatólia ▁xxmaj ▁ocidental ; ▁pre s ume - se ▁que ▁ele ▁tenha ▁perdido ▁todos ▁os ▁seus ▁cargos . ▁xxmaj ▁como ▁membro ▁de ▁uma ▁família ▁imperial ▁e ▁um ▁general ▁respeita do ▁e ▁vitorioso , ▁ele ▁não ▁teve ▁dificuldades ▁para ▁criar ▁um ▁grande ▁exército ▁quando ▁ele ▁resolveu ▁se ▁re bel ar ▁aberta mente ▁contra ▁o ▁novo ▁regime . ▁xxmaj ▁ va ta tz es ▁considerava ▁xxmaj ▁and r ônico ▁um ▁\" ad ver s ário ▁de mon íaco \" ▁que ▁estava ▁\" con centr ado ▁em ▁ex termina r ▁a ▁família ▁imperial \". ▁xxmaj ▁pelo ▁menos ▁esta ▁segunda ▁acusação ▁era ▁verdade . ▁xxmaj ▁and r ônico ▁i ▁enviou ▁o ▁general ▁xxmaj ▁and r ônico ▁xxmaj ▁ lam par das ▁( ou ▁xxmaj ▁la par das ) ▁contra ▁xxmaj ▁ va ta tz es ▁com ▁um ▁grande ▁exército . ▁xxmaj ▁ va ta tz es , ▁que ▁havia ▁fica do ▁seria mente ▁do ente , ▁deu - lhe ▁combate ▁perto ▁de ▁xxmaj ▁fila dé l fia . ▁xxmaj ▁ele ▁primeiro ▁instru iu ▁seus ▁filhos , ▁xxmaj ▁manuel ▁e ▁xxmaj ▁aleixo , ▁em ▁como ▁dis por ▁o ▁exército ▁e , ▁em ▁seguida , ▁foi ▁carregado ▁até ▁uma ▁colina ▁de ▁onde ▁ele ▁podia ▁observar ▁a ▁batalha ▁numa ▁ lit eira . ▁xxmaj ▁as ▁forças ▁de ▁xxmaj ▁ va ta tz es ▁foram ▁vitoriosa s ▁e ▁as ▁tropas ▁remanescente s ▁de ▁xxmaj ▁ lam par das , ▁em ▁fuga , ▁foram ▁persegui das . ▁xxmaj ▁porém , ▁uns ▁poucos ▁dias ▁depois , ▁em ▁16 ▁de ▁maio ▁de ▁11 82 , ▁xxmaj ▁joão ▁xxmaj ▁ va ta tz es ▁morreu . ▁xxmaj ▁sem ▁o ▁líder , ▁a ▁revolta ▁rapidamente ▁se ▁de s fe z ▁e ▁os ▁filhos ▁do ▁general ▁fugir am ▁para ▁a ▁corte ▁do ▁sultão ▁se l jú cida . ▁xxmaj ▁posteriormente , ▁quando ▁tentava m ▁chegar ▁à ▁xxmaj ▁sicília ▁por ▁mar , ▁na u frag aram ▁e ▁terminaram ▁presos . ▁xxmaj ▁eles ▁foram ▁então ▁ce gado s ▁por ▁ordem ▁de ▁xxmaj ▁and r ônico ▁xxup ▁i , ▁que ▁considerou ▁a ▁morte ▁de ▁xxmaj ▁ va ta tz es ▁como ▁\" pro vi dência ▁divina \". ▁xxmaj ▁depois ▁do ▁fracasso ▁da ▁revolta , ▁ele ▁se ▁en cora jou ▁a ▁declara r - se ▁co imperador ▁ao ▁lado ▁de ▁xxmaj ▁aleixo . ▁xxmaj ▁joão ▁xxmaj ▁com ne no ▁xxmaj ▁ va ta tz es ▁é ▁uma ▁das ▁poucas ▁figuras ▁cujo ▁caráter ▁é ▁descrito ▁com ▁uma ▁admira ção ▁ pura ▁nas ▁obras ▁do ▁historiador ▁bizantino ▁xxmaj ▁ nice tas ▁xxmaj ▁con ia tes ."
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.train_ds.x[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Backward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 38min 20s, sys: 25 s, total: 38min 45s\n",
      "Wall time: 20min 19s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = (TextList.from_folder(dest, processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "            .split_by_rand_pct(0.1, seed=42)\n",
    "            .label_for_lm()\n",
    "            .databunch(bs=bs, num_workers=1, backwards=True))\n",
    "\n",
    "data.save(f'{path}/{lang}_databunch_corpus2_100_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "nbpresent": {
     "id": "052df7c2-f57b-4596-9189-c9e01102e5e9"
    }
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "defab6d9-ba04-4943-b574-9300e20d5e1c"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.74 s, sys: 2.58 s, total: 6.32 s\n",
      "Wall time: 12.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15_multifit', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "670d6ea2-913e-4cd2-80d8-1e3d34f0b9c0"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.75 s, sys: 1.73 s, total: 6.48 s\n",
      "Wall time: 6.88 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, drop_mult=0., pretrained=False, \n",
    "                               metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of parameters: 46020150\n"
     ]
    }
   ],
   "source": [
    "print(f'number of parameters: {sum([parameter.numel() for parameter in learn.model.parameters()])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(15000, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(15000, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): QRNN(\n",
       "        (layers): ModuleList(\n",
       "          (0): QRNNLayer(\n",
       "            (linear): WeightDropout(\n",
       "              (module): Linear(in_features=800, out_features=4650, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (1): QRNN(\n",
       "        (layers): ModuleList(\n",
       "          (0): QRNNLayer(\n",
       "            (linear): WeightDropout(\n",
       "              (module): Linear(in_features=1550, out_features=4650, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (2): QRNN(\n",
       "        (layers): ModuleList(\n",
       "          (0): QRNNLayer(\n",
       "            (linear): WeightDropout(\n",
       "              (module): Linear(in_features=1550, out_features=4650, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (3): QRNN(\n",
       "        (layers): ModuleList(\n",
       "          (0): QRNNLayer(\n",
       "            (linear): WeightDropout(\n",
       "              (module): Linear(in_features=1550, out_features=1200, bias=True)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "      (3): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=15000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e74f8988-7c26-495b-90fb-0da012f54c1a"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e0333edf-f5cd-4c7d-9db3-01dfb7cc4bb8"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8dcnO2QDTNiCCCjigrLFveDeUlr32krrLVWrtdXWpfXe9vq71W63trbWra3i2mut9VLrrdYVrYqorCIQXAiyaAiQYMwG2ef7+2MmMMYAgcxZJvN+Ph7zmJkz58z3k2F45+R7vud7zDmHiIikjrSgCxAREX8p+EVEUoyCX0QkxSj4RURSjIJfRCTFZARdQE8UFRW5UaNGBV2GiEhSWbp06VbnXHHX5UkR/KNGjWLJkiVBlyEiklTMbEN3yz3r6jGz+82syszK4pb9zMxWmNlbZva8mQ33qn0REemel338DwLTuyy72Tl3pHNuIvBP4Mceti8iIt3wLPidc/OAmi7L6uOe5gI6bVhExGe+9/Gb2S+ArwN1wMm7We8y4DKAkSNH+lOciEgK8H04p3Pueufc/sDDwJW7WW+2c67UOVdaXPypg9IiIrKPghzH/xfgvADbFxFJSb4Gv5mNjXt6JvCun+2LiIiHffxm9ghwElBkZhXADcAMMxsHRIANwOVetS8iksy21Dfz5wUbOGdSCWOK8xL63p4Fv3NuZjeL7/OqPRGRvmTDR9u5419rOGb0fgkPfs3VIyISQvVNbQAU9Ev8/rmCX0QkhOqbY8Gfk5nw91bwi4iE0M49fgW/iEhKqG9uB6AgR109IiIpob6pjdysdDLSEx/TCn4RkRCqa2rzpJsHFPwiIqFU39zmyYFdUPCLiIRSfVO7J0M5QcEvIhJK2uMXEUkx9c1tFKqPX0QkddRt18FdEZGUEYk4GlraPRnDDwp+EZHQaWxtxzlvztoFBb+ISOjsmK5BB3dFRFJDfVNsugYN5xQRSQ07ZuZUV4+ISGqoU1ePiEhq6ezj1zh+EZEUsXNKZgW/iEhK6Nzjz9M4fhGR1FDf3EZ+TgbpaebJ+yv4RURCpq7JuwnaQMEvIhI60SmZFfwiIikjOiWzN/37oOAXEQmdeg8vuwgKfhGR0Globlcfv4hIKqlv8u4iLKDgFxEJlY7Oufg9mqANFPwiIqHS0OztPD2g4BcRCZWdUzIr+EVEUsKOKZk1nFNEJDXsuPqW9vhFRFJD5x6/RvWIiKSIOu3xi4iklh0Hd9XHLyKSGuqb20gzyM1S8IuIpIT6pjbyczJJ82gufvAw+M3sfjOrMrOyuGU3m9m7ZrbCzB43swFetS8ikozqm709axe83eN/EJjeZdlcYLxz7khgNfAjD9sXEUk6dR7P0wMeBr9zbh5Q02XZ88659tjTBcAIr9oXEUlG9R5ffQuC7eO/GHhmVy+a2WVmtsTMllRXV/tYlohIcKIXYemDwW9m1wPtwMO7Wsc5N9s5V+qcKy0uLvavOBGRAEUvu+htH7+3794NM5sFfBE41Tnn/G5fRCTM/Njj9zX4zWw68B/Aic657X62LSISdm0dEba3dnh61i54O5zzEeANYJyZVZjZJcCdQD4w18zeMrO7vGpfRCTZdE7Q5vWoHs/2+J1zM7tZfJ9X7YmIJLv65s65+JN3HL+IiOyFHVMy98VRPSIi8mk7LsKSrH38IiKyd3bOzKngFxFJCTvn4lcfv4hISvDj6lug4BcRCY36pjYy0ox+memetqPgFxEJifrmNgr6ZWLm3Vz8oOAXEQmN+qZ2Ty+52EnBLyISEp17/F5T8IuIhIQfF2EBBb+ISGjU+XARFlDwi4iExkeNrRTlZXnejoJfRCQEmts6qGtqozg/2/O2FPwiIiGwtbEFQMEvIpIqqhsU/CIiKWVH8OfleN6Wgl9EJASq1dUjIpJaOvf499OoHhGR1FDd0MKg3Cwy072PZQW/iEgIVDe0UJznfTcPKPhFREKhqqGFwQUKfhGRlKE9fhGRFOKco7qxxZcRPaDgFxEJXH1zO63tEQW/iEiq8POsXVDwi4gEbudZuwp+EZGU4OdZu6DgFxEJnLp6RERSTHVDC1npab5cdhEU/CIigatqaKY4Pxsz86U9Bb+ISMCqG1oo8qmbBxT8IiKB8/OsXVDwi4gEbquPZ+2Cgl9EJFDtHRE+2taq4BcRSRU121pxzr+hnKDgFxEJVJXPZ+2Cgl9EJFB+n7ULHga/md1vZlVmVha37HwzW2VmETMr9aptEZFkUV0fDf7BfSH4gQeB6V2WlQHnAvM8bFdEJGkEscef4dUbO+fmmdmoLsveAXw7O01EJOyqG1rIz8kgJzPdtzbVxy8iEqDqBn/H8EOIg9/MLjOzJWa2pLq6OuhyREQ84fdZuxDi4HfOzXbOlTrnSouLi4MuR0TEE35ea7dTaINfRCQVhLarx8wONLPs2OOTzOx7ZjZgD9s8ArwBjDOzCjO7xMzOMbMK4DjgKTN7rrc/gIhIstre2k5jS7vvwd/TUT2PAaVmdhBwH/AE8Bdgxq42cM7N3MVLj+9VhSIifVTnlbcG5+f42m5Pu3oizrl24BzgVufcNcAw78oSEen7/L7kYqeeBn+bmc0EZgH/jC3z5xphIiJ9VHUA8/RAz4P/IqL98r9wzq0zs9HAn70rS0Sk7wvirF3oYR+/c+5t4HsAZjYQyHfO3eRlYSIifd3mumYy0oxBuVm+ttvTUT0vm1mBmQ0ClgMPmNkt3pYmItK3VdY2MbQwh/Q0f6ex6WlXT6Fzrp7oBGsPOOemAKd5V5aISN9XWdfM8MJ+vrfb0+DPMLNhwJfZeXBXRER6obK2ieED/B3KCT0P/p8CzwHvO+cWm9kYoNy7skRE+raOiGNzXTPDB/i/x9/Tg7tzgDlxz9cC53lVlIhIX1fd0EJ7xDEsgODv6cHdEWb2eOyKWlvM7DEzG+F1cSIifVVlXRMAJSHu6nmA6DQNw4ES4MnYMhER2QeVtdHgD6Krp6fBX+yce8A51x67PQhormQRkX2UDMG/1cwuNLP02O1C4CMvCxMR6csqa5vJy86gIMf/2W96GvwXEx3KuRnYBHyJ6DQOIiKyD4Iaygk9DH7n3AfOuTOdc8XOucHOubOJnswlIiL7oLKuKZBuHujdFbiuTVgVIiIpprK2mWEBnLULvQt+fyeXEBHpI5rbOqjZ1hrIUE7oXfC7hFUhIpJCghzRA3s4c9fMGug+4A0IpmIRkSRXWdsMEFhXz26D3zmX71chIiKpYudZu8nXxy8iIvugsrYJMxhS6O+Vtzop+EVEfFZZ20RxXjbZGemBtK/gFxHxWWVtcyCzcnZS8IuI+KyyrimwoZyg4BcR8ZVzLjpdQ0AjekDBLyLiq4+3t9HcFlFXj4hIqug8eUtdPSIiKSLos3ZBwS8i4qvO4A/qrF1Q8IuI+KqyrpmsjDT2y80KrAYFv4iIj6IjenJISwtugmMFv4iIj6JX3gp2jksFv4iIj4K8AEsnBb+IiE/aOiJsaWgOdCgnKPhFRHyz8eMmnIMRA/sHWoeCX0TEJ6u3NABw0JC8QOtQ8IuI+KS8qhGAsYMV/CIiKWH1lgaGF+aQn5MZaB2eBb+Z3W9mVWZWFrdskJnNNbPy2P1Ar9oXEQmb8i2NjB0S/BVtvdzjfxCY3mXZD4EXnXNjgRdjz0VE+ryOiOP96kYODrh/HzwMfufcPKCmy+KzgD/FHv8JONur9kVEwuSDmu20tEf6/B5/d4Y45zYBxO4H72pFM7vMzJaY2ZLq6mrfChQR8ULniJ6DUzD4e8w5N9s5V+qcKy0uLg66HBGRXinvHMoZ8Ige8D/4t5jZMIDYfZXP7YuIBGL1lkZKBvQjLzsj6FJ8D/4ngFmxx7OAf/jcvohIIFZvaWBsCA7sgrfDOR8B3gDGmVmFmV0C3AScbmblwOmx5yIifVp7R4S11dtC0b8P4NnfHM65mbt46VSv2hQRCaMNNdtp7YgEfsZup9Ae3BUR6SvKt0SnagjLHr+CX0TEY2Ea0QMKfhERz62uamTEwH7khmBEDyj4RUQ8V76lITTdPKDgFxHxVOeInrAM5QQFv4iIp9Z/FB3Rc/Bg7fGLiKSE8hDN0dNJwS8i4qHVWxoxC8+IHlDwi4h4anVVA/sP7E+/rPSgS9lBwS8i4qG3PqjliJLCoMv4hHAMKvVIzbZWtrW0B13GHpnt7rVdv9j1lfhVrcurXd/GunnSuc0n3+eTdVjsddu5Uex5dJ341812Pk4zSLPOZbv5gUX6kI21TWysbeLSqaODLuUT+nTw/27uah5asCHoMqQbaQbpaYZZ9JdCuhlpZqSlGRlpRnrsPi3NyEpPIysjjcz0NDLTjZzM9Ngtjf5ZGeRlZ5CfE70V5WVTnJ9NUV42Iwb2C/yi1pLaFq+LXoTwqNGDAq7kk/p08J8zuYQJ+w/wvV3nXM/X3ccXXZcX45vsulnXcuK37XzNdbOy67qOi24Zv03nz+pc9H2j9598Hons3C7iHM45IrHHO+4jjo64+46Io73D0R5xtHVEaO+I3re0R2hp76ChuZ2mtg62t7TTGLtFuvm8Sgb04+AheYwbWsDJ44opHTWI9DT9xSH+WLS+hvzsDA4ZWhB0KZ/Qp4N/8siBTB45MOgyxAfOORpb2tna2Ep1QwvVDS2s/2gbq7c08N7mBuav2cpdr7zP4PxsZhwxjDMnDtd3Qzy3eF0NU0YNDN3ORp8OfkkdZkZ+Tib5OZmMLsr91OvbW9t58Z0qnlqxiUcWfcCDr6/n6FGDuOKUg5g2tkjHHSThPt7WSnlVI2dPKgm6lE9R8EtK6J+VwRkThnPGhOE0trQzZ8mHzJ63lln3L+LIEYVcfdpYTh43WL8AJGEWr4/1748KV/8+aDinpKC87AwuOmE0L193EjedewS129u4+MElfPWehaysqAu6POkjFq+vISs9jSNHhGsoJyj4JYVlZ6RzwdEjefH7J/KTMw/nvS0NnHHnfK7+6zKq6puDLk+S3KL1HzNh/0JyMsNz4lYnBb+kvMz0NGYdP4qXrzuJK04+kKfLNnPaLa/wt6UVezVCS6TT9tZ2Vm2sC2U3Dyj4RXYoyMnkus8dwrNXTWXc0Hx+MGc533hgMRtrm4IuTZLMsg9qaY+40I3f76TgF+liTHEej152HD8583AWr6/hs7e8wp9eX0+kuxMFRLqxaF0NZjDlgHAOGVbwi3QjLc2Ydfwonrt6GpMPGMgNT6ziS3e9vmOKXZHdWby+hkOHFlAQ0jPHFfwiu7H/oP78z8VHc8uXJ7B26zZm3P4qt79YTltHJOjSJKTaOiIs+6CWo0PazQMKfpE9MjPOnTyCF649kenjh3HL3NWc84fXeHdzfdClSQiVbayjqa0jtAd2QcEv0mNFedncMXMSd104mU21zZxxx3zu/Fc57dr7lzgvvVeNGRwzRsEv0mdMHz+MudeeyOcOH8pvnl/NV+9ZqJE/ssPTKzdx9KhBFOVlB13KLin4RfbBoNws7vzqZH73lQmsqqzj87fO45mVm4IuSwJWvqWBNVWNfOHIYUGXslsKfpFeOGfSCJ6+aiqji3L59sNv8l//V0Zru7p+UtVTKzdhBtMPHxp0Kbul4BfppQP2y2XO5cdz2bQxPLRgAxfet5CPGluCLksC8MzKzRx1wCAGF+QEXcpuKfhFEiArI43/nHEot10wkeUf1nLmna+xqlITvqWSNVWNvLelgRlHhHtvHxT8Igl11sQS5lx+HB0Rx5f++AZPq98/ZXT+W08fH+7+fVDwiyTckSMG8MR3T+CQYfl85+E3ufWF1ZruIQU8vXITpQcMZGhhuLt5QMEv4onB+Tk8cumxnDu5hFtfKOfKR95ke2t70GWJR9ZWN/Lu5gY+f0T49/ZBwS/imZzMdH57/gSun3Eoz5Zt5vy73gjNeP+W9g7e+rCW5raOoEvpE54p2wyQFP37oEsvinjKzLh02hgOGpLH9/6yjLPunM/d/zaFKQcEc1ZndUMLDy/cwJ8XfMDWxhZyMtM4/sAiTjlkMJ89bEjoR6OE1VMrNjF55ACGFfYLupQe0R6/iA9OHjeYx684nrzsDGbOXsicJR/6XsMtz7/HCTf9i1tfKGd8SQG/PX8CFxw1kjVVjfy//yvjM796iRufWEVVg64+tjdWb2ng7U31fOHI4UGX0mOWDFcYKi0tdUuWLAm6DJFeq93eypV/Wcb8NVu5+ITR/GjGIWSme7//tb21nfE3PMfUscX8+IzDOLA4b8drzjnWVDVy3/x1zFlaQWa68fXjRnH5iQcyKDfL89qS3U3PvMs9r65lwY9OpTg/XNM0mNlS51xp1+WB7PGb2VVmVmZmq8zs6iBqEAnCgP5ZPHjRUVx0wijuf20dX7tnoS972O9sqifi4MJjD/hE6EO0O2rskHxuOu9IXrz2RGaMH8Y9r65l2q9f4tYXVtPQ3OZ5fcmqI+J4fFkFJx1cHLrQ3x3fg9/MxgOXAkcDE4AvmtlYv+sQCUpGeho3nHE4t10wkRUba/ni7fNZuqHG0zZXVkRPJjuipHC3640qyuWWr0zk+aunMXVsEbe+UM60X7/EPfPW0tSqA8Fdvf7+VrbUt3Du5BFBl7JXgtjjPxRY4Jzb7pxrB14BzgmgDpFAnTWxhMe/cwL9stL5yt0L+MPLa+jwaLz/yo31FOVlM6SgZ3ulY4fk88cLp/DElScwvqSQXzz9DtNufokHX1unkUBxHltaQUFOBqceOjjoUvZKEMFfBkwzs/3MrD8wA9i/60pmdpmZLTGzJdXV1b4XKeKHQ4cV8MSVn+Gzhw/h18++x8zZC/iwZnvC2ynbWMcRJQWY2V5td+SIATx0yTE8etmxjCnK5cYn3+bk37zMXxd94NkvqWTR2NLOs6s288UJw8nJTA+6nL3ie/A7594BfgXMBZ4FlgOfOrPFOTfbOVfqnCstLi72uUoR/xT2y+T3X53Mb8+fwNub6plx26s8vqwiYe/f1NpBeVXDHrt5dueYMfvx18uO5eFvHsPQwhx++PeVnHHHfBau/ShhdSabp1duorktwnlJ1s0DAR3cdc7d55yb7JybBtQA5UHUIRIWZsZ5U0bwzFVTGTc0n2seXc5Vf11GfQIOrL4dO7A7vhfB31njCQcV8fdvH88dMydRu72Vr8xewBUPv0llSE5M89Pf36xgdFEuk0cOCLqUvRbUqJ7BsfuRwLnAI0HUIRI2+w/qz6PfOo7vn34w/1yxiRm3vcqS9b078Fu2MXZgd0Tvgr+TmXHGhOG8+P2TuOa0g3nx3S2cdssr3P3K+ylzEfqKj7ezYG0N504q2evuszAI6gSux8zsbeBJ4Arn3McB1SESOulpxndPHcucy4/DDL589xvcMnf1Pl/bd+XGOoryshia4LNy+2Wlc9VpY5l7zYkcf2ARv3zmXWbc9ioLUqD755FFHwBw9qSSgCvZN0F19Ux1zh3mnJvgnHsxiBpEwm7yyIE8/b2pnD2phNtfLOfLd7/BBx/t/YHfso11jC8p9GzPdP9B/bl3Vin3fr2UprYOLpi9gGsffYvqhr55MZplH3zM3a+s5YwJw9l/UP+gy9knmrJBJMTyczK55csTuX3mJMqrGplx+6s8trSCnp5x39zWQXlVY68O7PbUaYcNYe41J3LlyQfx5IpKTvntyzz0xvo+NfqnvrmN7/11GUMKcvj52eODLmefKfhFksCZE4bzzFVTOWxYAd+fs5zLHlpKVf2ez/h9e1M9HRHX6wO7PdUvK50ffG4cz149jSNHFPJf/1jFF25/lZfeq+rxL6uwcs5x/eNlVNY2c/vMiRT2ywy6pH2m4BdJEiMG9ueRy47l+hmHMm91Naf/bh5/f3P3e/87Duz6FPydDizO48+XHMPvvzqZ7a0dXPTAYmbes4DlH9b6WkcizVlawZPLK7nmtLGBza6aKAp+kSSSnhad5vmZq6YydnAe1/7vci56cPEuT/paWVHHfrlZDAvgqlBmxheOHMYL157IT848nPItjZz1+9f4t/sWMr98a1L9BbB0w8fc8I9VHDdmP7590kFBl9NrCn6RJDSmOI9Hv3UcP/7iYSxaV8Ppv+t+OOVKjw/s9kRWRhqzjh/Fy9edxHWfG8e7mxu48L6FfOH2+Ty+rCL0Q0Bffq+KC+9dyJCCbG69YCLpack3fLMrBb9IkkpPMy7+zGjmXnsinzmomF8+8y5n3DF/x7h/Pw/s9kR+TiZXnHwQ8//jZH513hG0tHdwzaPLmfqrl7jrlfepawrfLKD/eGsj3/zTEkYX5TLn8uMZ0kcuVKP5+EX6iOdWbebGJ1axqa6ZcyaVMH38UL710FLuunAK08eH75KAkYjjldXV3PPqWl5//yNys9I5d/IILjz2AMYNzQ+0Nucc97+2np8/9TZHjRrEvbNKKchJvoO5u5qPX8Ev0odsb23n9y+t4Z5562iNdaG89sNTKBkQ7ksCrqqs4/7563lyRSWt7RGOHj2Irx0zktMOHUJutr9XiN1U18S//20Fr5Zv5fTDhnDHzElJNwlbJwW/SApZv3UbP3/qHWq3t8bOAE6Ofumaba3MWfIhf164gQ9rmsjOSOPEg4v5/BFDOXncYAb09+6KYM45/v7mRm58chXtHY7/nHEIXzvmANKSuE9fwS8iSSMScSxaX8OzZZt5tmwzm+ubMYNDhhZwzOhBHDtmEEeOGMCwwpxe/1Jr74jwTNlm7p2/juUf1nLUqIHc/KUJjCrKTdBPExwFv4gkpUjE8VZFLa+Vb2XhuhqWbKihuS3ajTUoN4vDhxdw8JB8hg/ox7DCHIYW5jCkIIeivCyyMz7dRROJOCrrmlhT1UjZxjoeWfQhG2ubGF2Uy6VTx/CVo/bvEyN3QMEvIn1Ea3uEsso6yjbWsWpjPWWVdbxf3bjjl0G8wn6ZOy4Y3xFxdEQcNdtaaYq7itixYwbxzc+M4ZRDBid1t053dhX8/h41ERHppayMNCaPHMjkkQN3LHPOUdfURmVtM5vqmqhuaKGqoYXqhhZqtreSZka6QVqaMaBfFgcNzuOgwXkcWJzLfnnJc5H0RFHwi0jSMzMG9M9iQP8sDhteEHQ5oacTuEREUoyCX0QkxSj4RURSjIJfRCTFKPhFRFKMgl9EJMUo+EVEUoyCX0QkxSTFlA1mVg3UAnVdXircw7I9Pe68LwK27kNp3bW/N/Xtqebuao1/fV/q3pead1dXd8+7q7U3n7WfNcc/Dvv3Iyw1d7dc34898+P7McA5V/ypd3bOJcUNmL23y/b0OO5+SaJqSmTNu6g1ft29rntfat5dXT35fHv7WftZczJ9P8JSs74f4f9+dL0lU1fPk/uwbE+Pu9u+tzXt6fW9qTn+eZA1d7d8d8+7q7U3dftZc/zjsH8/wlJzd8v1/dgzP78fn5AUXT1eM7MlrpsZ7MIuGetWzf5IxpohOetOxpqTaY/fS7ODLmAfJWPdqtkfyVgzJGfdSVez9vhFRFKM9vhFRFKMgl9EJMX0ueA3s/vNrMrMyvZh2ylmttLM1pjZ7RZ3FWcz+66ZvWdmq8zs12Gv2cxuNLONZvZW7DYjkTV7VXfc6z8wM2dmRYmr2LPP+mdmtiL2OT9vZsOToOabzezdWN2Pm9mAJKj5/Nj/v4iZJexgam9q3cX7zTKz8thtVtzy3X7nfbUv40/DfAOmAZOBsn3YdhFwHGDAM8DnY8tPBl4AsmPPBydBzTcCP0i2zzr22v7Ac8AGoCjsNQMFcet8D7grCWr+LJARe/wr4FdJUPOhwDjgZaA06FpjdYzqsmwQsDZ2PzD2eODufq4gbn1uj985Nw+oiV9mZgea2bNmttTMXjWzQ7puZ2bDiP4HfsNF/5X+Bzg79vK3gZuccy2xNqqSoGbPeVj374B/BxI+8sCLmp1z9XGr5ia6bo9qft451x5bdQEwIglqfsc5914i6+xNrbvwOWCuc67GOfcxMBeYHvT/1a76XPDvwmzgu865KcAPgD90s04JUBH3vCK2DOBgYKqZLTSzV8zsKE+rjeptzQBXxv6Uv9/MBuKPXtVtZmcCG51zy70uNE6vP2sz+4WZfQh8Dfixh7V2SsT3o9PFRPdAvZbImr3Wk1q7UwJ8GPe8s/6w/FxAClxs3czygOOBOXFdatndrdrNss49twyif7YdCxwF/K+ZjYn95k64BNX8R+Bnsec/A35L9D+4Z3pbt5n1B64n2g3hiwR91jjnrgeuN7MfAVcCNyS41J2FJKjm2HtdD7QDDyeyxk8VksCavba7Ws3sIuCq2LKDgKfNrBVY55w7h13XH/jPFa/PBz/Rv2pqnXMT4xeaWTqwNPb0CaJBGf/n7gigMva4Avh7LOgXmVmE6MRM1WGt2Tm3JW67e4B/elRrvN7WfSAwGlge+w83AnjTzI52zm0Oac1d/QV4Cg+DnwTVHDvw+EXgVK92YuIk+nP2Ure1AjjnHgAeADCzl4FvOOfWx61SAZwU93wE0WMBFQT/c+0U1MEFL2/AKOIO1ACvA+fHHhswYRfbLSa6V9958GVGbPnlwE9jjw8m+qechbzmYXHrXAP8NRk+6y7rrCfBB3c9+qzHxq3zXeBvSVDzdOBtoNiL74WX3w0SfHB3X2tl1wd31xHtIRgYezyop995v26BNOrpDwSPAJuANqK/ZS8huhf5LLA89mX/8S62LQXKgPeBO9l5ZnMW8OfYa28CpyRBzQ8BK4EVRPekhiWyZq/q7rLOehI/qseLz/qx2PIVRCfGKkmCmtcQ3YF5K3ZL9EgkL2o+J/ZeLcAW4Lkga6Wb4I8tvzj2+a4BLtqb77xfN03ZICKSYlJlVI+IiMQo+EVEUoyCX0QkxSj4RURSjIJfRCTFKPglKZlZo8/t3WtmhyXovTosOpNnmZk9uaeZMc1sgJl9JxFti4CuwCVJyswanXN5CXy/DLdz0jJPxdduZn8CVjvnfrGb9UcB/3TOjfejPun7tMcvfYaZFZvZY2a2OHY7Ibb8aDN73cyWxe7HxZZ/w8zmmNmTwPNmdpKZvWxmf7PoXPUPd86ZHlteGnvcGJuUbbmZLTCzIbHlB8aeLzazn/bwr5I32DlBXZ6ZvWhmb1p03izBGF0AAAJTSURBVPazYuvcBBwY+yvh5ti618XaWWFmP0ngxygpQMEvfcltwO+cc0cB5wH3xpa/C0xzzk0iOnPmf8dtcxwwyzl3Suz5JOBq4DBgDHBCN+3kAguccxOAecClce3fFmt/j/OwxOapOZXomdUAzcA5zrnJRK8B8dvYL54fAu875yY6564zs88CY4GjgYnAFDObtqf2RDqlwiRtkjpOAw6Lm1GxwMzygULgT2Y2luiMiJlx28x1zsXPxb7IOVcBYGZvEZ3DZX6XdlrZOendUuD02OPj2DnH+l+A3+yizn5x772U6JztEJ3D5b9jIR4h+pfAkG62/2zstiz2PI/oL4J5u2hP5BMU/NKXpAHHOeea4hea2R3AS865c2L95S/Hvbyty3u0xD3uoPv/I21u58GxXa2zO03OuYlmVkj0F8gVwO1E5/IvBqY459rMbD2Q0832BvzSOXf3XrYrAqirR/qW54nOhQ+AmXVOq1sIbIw9/oaH7S8g2sUEcMGeVnbO1RG9VOMPzCyTaJ1VsdA/GTggtmoDkB+36XPAxbF54zGzEjMbnKCfQVKAgl+SVX8zq4i7XUs0REtjBzzfJjqdNsCvgV+a2WtAuoc1XQ1ca2aLgGFA3Z42cM4tIzoD5AVEL4ZSamZLiO79vxtb5yPgtdjwz5udc88T7Up6w8xWAn/jk78YRHZLwzlFEiR2BbEm55wzswuAmc65s/a0nYjf1McvkjhTgDtjI3Fq8fhSlyL7Snv8IiIpRn38IiIpRsEvIpJiFPwiIilGwS8ikmIU/CIiKeb/A9At74k85dAOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "55299290-8c05-415f-8f73-a3656c1d488c"
    }
   },
   "outputs": [],
   "source": [
    "lr = 3e-3\n",
    "lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "debf9e8a-b4ec-4b06-8344-a8f01150d941"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.712833</td>\n",
       "      <td>3.801982</td>\n",
       "      <td>0.689854</td>\n",
       "      <td>0.310145</td>\n",
       "      <td>44.789921</td>\n",
       "      <td>45:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.495445</td>\n",
       "      <td>3.612981</td>\n",
       "      <td>0.673845</td>\n",
       "      <td>0.326156</td>\n",
       "      <td>37.076435</td>\n",
       "      <td>44:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.416562</td>\n",
       "      <td>3.550838</td>\n",
       "      <td>0.667659</td>\n",
       "      <td>0.332342</td>\n",
       "      <td>34.842659</td>\n",
       "      <td>45:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.346702</td>\n",
       "      <td>3.473753</td>\n",
       "      <td>0.658722</td>\n",
       "      <td>0.341275</td>\n",
       "      <td>32.257648</td>\n",
       "      <td>45:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.269586</td>\n",
       "      <td>3.402409</td>\n",
       "      <td>0.649641</td>\n",
       "      <td>0.350360</td>\n",
       "      <td>30.036201</td>\n",
       "      <td>45:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.206794</td>\n",
       "      <td>3.321666</td>\n",
       "      <td>0.639204</td>\n",
       "      <td>0.360796</td>\n",
       "      <td>27.706499</td>\n",
       "      <td>45:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.124528</td>\n",
       "      <td>3.235740</td>\n",
       "      <td>0.627807</td>\n",
       "      <td>0.372192</td>\n",
       "      <td>25.425192</td>\n",
       "      <td>45:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.092082</td>\n",
       "      <td>3.157651</td>\n",
       "      <td>0.616111</td>\n",
       "      <td>0.383889</td>\n",
       "      <td>23.515293</td>\n",
       "      <td>45:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.963153</td>\n",
       "      <td>3.095485</td>\n",
       "      <td>0.606017</td>\n",
       "      <td>0.393983</td>\n",
       "      <td>22.097927</td>\n",
       "      <td>45:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.998470</td>\n",
       "      <td>3.079903</td>\n",
       "      <td>0.603129</td>\n",
       "      <td>0.396870</td>\n",
       "      <td>21.756315</td>\n",
       "      <td>45:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdDklEQVR4nO3de3xV5Z3v8c+zL7mHEAJCuCYgKmBpwEihKAftTRGt5yXHQ2vHGTozTLWOl9Npi/qqVWundmbsqGdGrTq2p0q1ivV4ar3SSrHFSxOlGALIHcItIeZK7slz/tgryd5hJ9nBvfZeJN/367Vf+1mX/azfXpt8XT57rbWNtRYREfEuX7ILEBGRgSmoRUQ8TkEtIuJxCmoREY9TUIuIeFzAjU5zcsfYmTOmu9G1iMiwVFpaetxaOy7aMleC+oyJUygpKXGjaxGRYckYs7+/ZRr6EBHxOAW1iIjHKahFRDzOlTFqEZGhaG9vp6KigpaWlmSX4rq0tDQmT55MMBiM+TWuBLXuHiIiQ1FRUUF2djYFBQUYY5JdjmustVRXV1NRUUFhYWHMr3Nn6ENJLSJD0NLSQl5e3rAOaQBjDHl5eUP+PweNUYuIJwz3kO52Ku9TQS0i4nEKahEZ8Wpra3nooYeG/Lply5ZRW1vrQkWRXApqDVKLyOmjv6Du7Owc8HUvv/wyo0ePdqusHjo9T0RGvDVr1rB7926KiooIBoNkZWWRn5/P5s2bKS8v58orr+TgwYO0tLRw0003sXr1agAKCgooKSmhsbGRSy+9lAsuuIBNmzYxadIkXnzxRdLT0+NSn4JaRDzlrt9spfxwfVz7nD1xFN+/fE6/y++9917KysrYvHkzGzZs4LLLLqOsrKznFLonnniCMWPG0NzczPnnn89VV11FXl5eRB87d+7k6aef5rHHHuPqq6/m+eef52tf+1pc6td51CIifSxYsCDiPOcHH3yQF154AYCDBw+yc+fOk4K6sLCQoqIiAM477zz27dsXt3p0RC0injLQkW+iZGZm9rQ3bNjA+vXrefvtt8nIyGDp0qVRz4NOTU3tafv9fpqbm+NWj876EJERLzs7m4aGhqjL6urqyM3NJSMjg+3bt/POO+8kuDodUYuIkJeXx+LFizn33HNJT09n/PjxPcsuueQSHnnkEebOncvZZ5/NwoULE16fsTb+I8ozZs+1u8u3xL1fERmetm3bxqxZs5JdRsJEe7/GmFJrbXG09TX0ISLicbopk4iIx+mIWkTE4xTUIiIe50pQa+RDRCR+dEQtIuJxCmoRkVOQlZUFwOHDh1mxYkXUdZYuXUpJSckn3paCWkTkE5g4cSLr1q1zdRu6MlFEBPjud7/LtGnTuP766wG48847McawceNGampqaG9v55577uHLX/5yxOv27dvH8uXLKSsro7m5mVWrVlFeXs6sWbPidr8PBbWIeMsra+Doh/Htc8Kn4NJ7B1xl5cqV3HzzzT1B/eyzz/Lqq69yyy23MGrUKI4fP87ChQu54oor+v3dw4cffpiMjAy2bNnCli1bmD9/flzKV1CLiADz5s2jsrKSw4cPU1VVRW5uLvn5+dxyyy1s3LgRn8/HoUOHOHbsGBMmTIjax8aNG7nxxhsBmDt3LnPnzo1LbQpqEfGWQY583bRixQrWrVvH0aNHWblyJWvXrqWqqorS0lKCwSAFBQVRb3Eazo1fU9d51CIijpUrV/LMM8+wbt06VqxYQV1dHWeccQbBYJA333yT/fv3D/j6JUuWsHbtWgDKysrYsiU+N6fTEbWIiGPOnDk0NDQwadIk8vPzueaaa7j88sspLi6mqKiIc845Z8DXX3fddaxatYq5c+dSVFTEggUL4lKXK7c5LZw11+7dptucikhsdJvTONzm1BhzizFmqzGmzBjztDEmLQ61iohIDAYNamPMJOBGoNhaey7gB1YO+CINUouIxE2sXyYGgHRjTADIAA67V5KIjERuDMN60am8z0GD2lp7CPg34ABwBKiz1r7edz1jzGpjTIkxpqSxsXHIhYjIyJWWlkZ1dfWwD2trLdXV1aSlDW30eNAvE40xucDzwP8EaoHngHXW2qf6e03BOZ+y+7bH+coiERm22tvbqaioGPQc5eEgLS2NyZMnEwwGI+YP9GViLKfnfR7Ya62tcjr7NfBZoN+gFhEZimAwSGFhYbLL8KxYxqgPAAuNMRkmdMnN54Bt7pYlIiLdYhmjfhdYB7wPfOi85lGX6xIREUdMVyZaa78PfN/lWkREJAr9cICIiMcpqEVEPE5BLSLicQpqERGP0/2oRUQ8TkfUIiIe505Q65BaRCRudEQtIuJxCmoREY9TUIuIeJyCWkTE4xTUIiIep/OoRUQ8TkfUIiIep6AWEfE4BbWIiMcpqEVEPE5BLSLicQpqERGP002ZREQ8TkfUIiIe59IFLzqkFhGJF1eCuks5LSISN64EdXNbpxvdioiMSK4E9Ym2Dje6FREZkVwJ6qzUgBvdioiMSC6NUWuQWkQkXtwJ6i43ehURGZlcCepOnfYhIhI3rgR1hw6pRUTixrVfeGnrUFiLiMSDa5eQN+kUPRGRuHAtqE/oohcRkbhw74i6VUfUIiLx4FpQl+yvcatrEZERxbWgHpuV6lbXIiIjSkxBbYwZbYxZZ4zZbozZZoxZNNhr9GWiiEh8xHpTjgeAV621K4wxKUDGYC/QHfREROJj0KA2xowClgB/A2CtbQPaBnvdR8caP2ltIiJCbEMf04Eq4GfGmA+MMY8bYzL7rmSMWW2MKTHGlAA88ae9cS5VRGRkiiWoA8B84GFr7TzgBLCm70rW2kettcXW2uI41ygiMqLFEtQVQIW19l1neh2h4BYRkQQYNKittUeBg8aYs51ZnwPKXa1KRER6xHrWxz8Ca50zPvYAq9wrSUREwsUU1NbazcCQx54/PtHGmMyUIRclIiK9XLkyMSUQ6rayocWN7kVERhRXgnqcc/n4L9894Eb3IiIjiitB7fcZAH7x9n43uhcRGVFcCeqc9KAb3YqIjEiu3T1PRETiQ0EtIuJxrgf1pl3H3d6EiMiw5npQ/+iV7W5vQkRkWHMtqP/mswUAfHiozq1NiIiMCK4F9e2XzXKraxGREcW1oA76e7s+oV8kFxE5ZQk56+PFzYcTsRkRkWEpIUF92wsfJmIzIiLDkqtBXX73l3rajRr+EBE5Ja4GdUZK711Uz/3+a25uSkRk2HJ96CM/J83tTYiIDGuuB/Vvb7ywp73v+Am3NyciMuy4HtThv/Cy9N82uL05EZFhJyFnfTx+be+veOmoWkRkaBIS1J+fPb6nraNqEZGhSdhtTu9YPrunXbq/JlGbFRE57SUsqL9+QWFP+6qHN2GtTdSmRUROawn94YCCvIyeduGtLydy0yIip62EBvWGb18UMb1pt35UQERkMAn/Ka69P1rW0/7qY+8mevMiIqedhAe1MSZiumDNbxNdgojIaSUpP26764eXRkx/dKwhGWWIiJwWkhLUAb+PjWHj1V/89400tenueiIi0SQlqAGm5mVEnFs9+47X6OjsSlY5IiKelbSghshzqwHOvP2VJFUiIuJdSQ1qiDwLBOC//rg3SZWIiHhT0oPaGMOvr/9sz/QPXipXWIuIhEl6UAPMn5rL6iXTe6Z/8FI5dU3tSaxIRMQ7PBHUALctm8WFM8f2TH/67tf5u//z5yRWJCLiDZ4JaoAn//YzEdPrt1Wyq1LnWIvIyOapoAbYd+9lEdOf/8lGXtt6NEnViIgkX8xBbYzxG2M+MMa85GZBADvuuSRi+h+eLKWqodXtzYqIeNJQjqhvAra5VUi41ICftX8XOQxy/g/X09mle1iLyMgTU1AbYyYDlwGPu1tOr8VnjmXPP0eeYz3jtpd5ruRgokoQEfGEWI+o7we+A/R7jbcxZrUxpsQYU1JVVRWf4nyG576xKGLet9dt0R33RGREGTSojTHLgUprbelA61lrH7XWFltri8eNGxe3As8vGMNb37nopPkFa35LzYk23R9ERIa9WI6oFwNXGGP2Ac8AFxtjnnK1qj6mjMk4aRgEYN4P3uDM21+hpb0zkeWIiCTUoEFtrb3VWjvZWlsArAR+b639muuV9eHzGfbdexkXnX3y0fo533uVlY++3fuDufrhXBEZRgLJLmCoHru2mKf/fJDv/d+yiPnv7PmYwltf5vFri7mobA3+ivcgdxqMngqjp0W2R00Enz9J70BEZGiMdeHos7i42JaUlMS933DNbZ3MuuPVqMtW+n/PNwqOUeA/DrUHoP4wEPY+fUHImRwK7txpofDuCfNpkHUG9PnJMBERNxljSq21xVGXna5B3W1XZSOf/8kfBlznwsJsvvWZDIqy66FmP9TuDwV4d/tEn7NUAmnRj8S7gzw9V0EuInE1rIMaYPvRejbtqubul8pjWn/V4gJuvHgmuZkpoRltJ6D2YCi0e4K8u30AWmojO0jJjh7g3UfoqdlxfociMtwN+6AOV3aojuX/+49Dft2i6Xm8vaeaGz83k9fKjvLqzRf2/mJ6S11vaEeEuXNU3n4isrP0MU6IT4HUHEjJgGA6BDOjtDND093tlAwIOo9Aqo7cRUaIERXUAL/bdoyn3zvI+m3H4tLfff/j0+TnpGGMYePOKq6aP5kZ4zIBMABN1dEDvK4C2hpDR+ztzdA5xPuVGN/JQX5SqA8Q+t3tlKzQUX74cyAlLvtGROJjxAV1uPqWdhpbOqhubOPy/xj6kfYn9dA182loaeeSOfnkpBqqPq5hbGoHG7ceYGKGpbOtkRk5PtpaGsk0bU6oN4Uebd3PTtD3tPsud567hvBjC/5USO0O7uywdpbTHhXW7p4/KjTdHfg9oa8jf5FPakQHdX9O98vQJ41O53vLZ/GNp94H4MVvLqa8oppN2w5yzfyx+Dqa2Xukkpm5PuZPCEJrY+jovrXBaTeEtbvnN0Su03dIpz++oBPgTnhHhHnY/JTM3mUpmc5z1snzA6ku7jkRb1JQx6ito4uPjjXQ2tHJ9Wvf51j98Ly16pVFE8kfnc7DG3bzhdnjmT4uk/Om5rL6yVIev7aYBdPH8ONXtrNk5hhyfG3k+FsYl9LOe9v3MSfPz/4jlRSN9zHKtPYEfldrI762RmxrA10tDfjbG0NhHx7+xPhvzRfsDfLU7lDPDAV+d3vAwO8zLyUL/KfdJQMywiioPyFrLXXN7by18zhnjc+mdH8Nt73wYbLLOi38/YWFPPbWXi47dwL/efU5oaGbNifE2044j4bedmtv27Y1YHrmN/a+tmfcvyn2QgJpYeE/qnfoJuIRbX74vKzQ2L/Pc7+3IcOAgjrBPqyoY3dVIzf/anPPvEvmTOCCmWM5e0I2d/+mnKvPn8J9r++gtqmdGeMy2V0VGmYI+g0PX3Me5Ufq+ckbHyXrLXhWdlqAqWMy2Hq4njHpPr61dDIPvPIXMk0LN14wgXe2H+CCqem0N9czIwf2Halk3hkBaG9iTLCVtK4WAu2N0FrfO9zT/ehojqEC00/Ixxr23cNA2TrKlwgK6tNcdWMrmakB0oKhy963H61nf3UTX5w9nrJD9fh8sL68kk9PyaGzy3LhzHGkBHx8cKCGTbur+dfXdiT5HXjLX+74In6/4cXNh7j35e18cc4EvjB7PDc89S5Xf2o0hdldNDXUcOG0NEp3HKC+7mPOzw8wMb2DpvoaGuprmJrVRV6glbYTddDWQDbNtDfVkdbVFPswTzAT0kaFgjxtVG+o98zLiZxOzT55mc7eGTYU1AKEAv6M7DRSAz5a2jt5/0AtsyeOormtgwd/t4vrls7gWH0Li88cS0NLB+/t/Zidxxq4r8+R/Vnjs/joWGPEvOVz83lpy5FEvh3PMnSRQStZNJNlmsl2nrNoJts08Q8LxnJmDjQ31uBvqyfY3ohxjvCbG2sItDcQbG+MbWgnkNZP0OcM8B+BnLB52aFz9nXWTtIpqCUpqhpaue/1HfzgynOZefsrEctSAz5aO6LfS9xnYLj/6tq0vAz2Vw8cxAE6esJ9lPOcRTPZNJFtmsjuWdbUu8w0Mz27E9tSR0rHCbJNDMM5xhflFM3u6VF9prOJOLun77RC/5QpqGVYOFLXTFZqgOy0IF1dltIDNZxo7WDJzHHsrGyk+kQrn50xlgfW72Tp2eP4w0dVFBfkUt/cwYGPT/DPL29P9ltIOB9dPeEeOrpvItt5njfez6risdDagG1toKmhljTb1HP2Dq0NmLZGbEvozB2fHfy+79b46Apm4U8bFSX4s2kPZNKVkkVqRk7vqZvdV+OmZPZelRs+zx9MwJ5KPgW1CKGLn7659n1Wnj+VnZUNfPUzUzkjOw2A9w/UkOL30WUtfzlYy9kTRhHwG+ZPzeXVsqOMyUzhvGm5FN39Og0tHdy27Bze3F7F3Mk5/HTjHu66Yg5pQR+TRmfw1q4qPm5s47nSiiS/43iypNFGFi1kmSYyaSHbNJNJszPE00IWzWQ6Qz2ZtPQM92SZ0PS0rE5aT9SRRTN+M4Tc8QX6uRI3SqgH0/u0B1s30zNf6iqoRTxix9EGvnT/xgHXuXbRNH7x9v4EVZQModDPdoL9ny6awtKCTN7ecZAM00r5/qP81Xlj2XvkOB/X1DIz10ews5kdFceYlm3oajtBReVxJmZYGhvqyQm0k5/eRVNTA6ldLQTMEH+ez58Sehhf6OHzO22/0/aHhnN62r7B5/t8ffrwDTrfXPmQglrES6oaWhmblUJrRxcNLR2My45+Neah2mbyMlN6zvgJd7yxlbJDdcyZmENbZxeTRqdjraWts4sUvw9jDK9vPcrqJ3t/7rTsri/R2WX59F2vu/beksuSQgfptJJOKxnGeaaV9LB2hmklLax90ZmjmTE2Ax9ddHZ2YLA0tbSRGYSAAWwndHWC7Qpr2552dUMzGSmGdD/OOl3OOp1h7d753dvwOcs7OzsIfHuHglpEeh1vbKX4nvUATBmTzn+fN5m2ji4e+cNuAP7bWeMI+n09Nzb705qLWXzv75NW70iw/8fLFdQikli1TW0AjM4Inetd19RO+ZF6Fs3I40hdM4t+pOAPN1BQe2MUXUSGne6A7paTEWTRjDwA8nPSef66RdQ1t3PDLz+gqa2Tv7+wkDGZqSyfm89vthxmdv4oTrR2Urq/hrHZKXx1wVSWPfAWo9KDfP2CQh7duIdvfeEsUgI+1m+rpGhKDkvOGkd+TjoPrN9J/ug0MlL83PDLD077Uz51RC0iI9pdv9nKz/60LyHbOm9aLqX7axidEaS2qZ1frV7IPz79AZUNrRr6EBGJp7aOLoJ+0/srUHEw0Ol5GvoQERmilEBi76Co+zWKiHicglpExOMU1CIiHqegFhHxOAW1iIjHKahFRDxOQS0i4nEKahERj1NQi4h4nIJaRMTjFNQiIh6noBYR8TgFtYiIxymoRUQ8btCgNsZMMca8aYzZZozZaoy5KRGFiYhISCz3o+4AvmWtfd8Ykw2UGmPesNaWu1ybiIgQwxG1tfaItfZ9p90AbAMmuV2YiIiEDGmM2hhTAMwD3nWjGBEROVnMQW2MyQKeB2621tZHWb7aGFNijCmpqqqKZ40iIiNaTEFtjAkSCum11tpfR1vHWvuotbbYWls8bty4eNYoIjKixXLWhwH+C9hmrf2J+yWJiEi4WI6oFwN/BVxsjNnsPJa5XJeIiDgGPT3PWvtHwCSgFhERiUJXJoqIeJyCWkTE4xTUIiIep6AWEfE4BbWIiMcpqEVEPE5BLSLicQpqERGPU1CLiHicglpExOMU1CIiHqegFhHxOAW1iIjHKahFRDxOQS0i4nEKahERj1NQi4h4nIJaRMTjFNQiIh6noBYR8TgFtYiIxymoRUQ8TkEtIuJxCmoREY9TUIuIeJyCWkTE4xTUIiIep6AWEfE4BbWIiMcpqEVEPE5BLSLicQpqERGPU1CLiHicglpExOMU1CIiHqegFhHxOAW1iIjHxRTUxphLjDE7jDG7jDFr3C5KRER6DRrUxhg/8J/ApcBs4CvGmNluFyYiIiGxHFEvAHZZa/dYa9uAZ4Avu1uWiIh0C8SwziTgYNh0BfCZvisZY1YDq53JVmNM2ScvzxVjgePJLqIfXq4NvF2fl2sDb9en2k5dPOub1t+CWILaRJlnT5ph7aPAowDGmBJrbXHM5SWQajt1Xq7Py7WBt+tTbacuUfXFMvRRAUwJm54MHHanHBER6SuWoP4zMNMYU2iMSQFWAv/P3bJERKTboEMf1toOY8wNwGuAH3jCWrt1kJc9Go/iXKLaTp2X6/NybeDt+lTbqUtIfcbak4abRUTEQ3RlooiIxymoRUS8zlobtwdwCbAD2AWsiWffUba1D/gQ2AyUOPPGAG8AO53nXGe+AR506toCzA/r56+d9XcCfx02/zyn/13Oa80g9TwBVAJlYfNcr6e/bcRQ253AIWf/bQaWhS271dnODuBLg32+QCHwrlPDr4AUZ36qM73LWV4QpbYpwJvANmArcJNX9t0AtXll36UB7wF/ceq761T7jFfdMdT2c2Bv2L4rSsbfhLOeH/gAeMkr+63ffIljcPqB3cB0IMX5gGbHq/8o29sHjO0z71+6dwqwBvix014GvOL8Y1gIvBv2ge5xnnOddncgvAcscl7zCnDpIPUsAeYTGYau19PfNmKo7U7gn6KsO9v57FKdf1S7nc+2388XeBZY6bQfAa5z2tcDjzjtlcCvomwvH+ePEsgGPnJqSPq+G6A2r+w7A2Q57SChAFg41D7jWXcMtf0cWBHlvST0b8JZ9r+AX9Ib1Enfb/3mSxyDcxHwWtj0rcCt8eo/yvb2cXJQ7wDyw/7IdjjtnwJf6bse8BXgp2Hzf+rMywe2h82PWG+AmgqIDEPX6+lvGzHUdifRwybicyN0ts+i/j5f54/kOBDo+++g+7VOO+CsN9j/mbwIfMFL+y5KbZ7bd0AG8D6hq4aH1Gc8646htp8TPagT+rkSuh7kd8DFwEun8lm4vd/CH/Eco452qfmkOPbflwVeN8aUOpevA4y31h4BcJ7PGKS2geZXRJk/VImop79txOIGY8wWY8wTxpjcU6wtD6i11nZEqa3nNc7yOmf9qIwxBcA8Qkdfntp3fWoDj+w7Y4zfGLOZ0NDWG4SO5IbaZzzr7rc2a233vvuhs+/+3RiT2re2GGv4pJ/r/cB3gC5n+lQ+C1f2WzTxDOqYLjWPo8XW2vmE7ur3TWPMkgHW7a+2oc6PFy/U8zAwAygCjgD3uVBbzHUbY7KA54GbrbX1/Zed+H0XpTbP7Dtrbae1tojQEeICYNYp9OnKPu1bmzHmXEJHlucA5xMazvhunGsblDFmOVBprS0Nnz1Afwndb9HEM6gTeqm5tfaw81wJvEDoH+kxY0w+gPNcOUhtA82fHGX+UCWinv62MSBr7THnD6kLeIzQ/juV2o4Do40xgT7zI/pylucAH/etxRgTJBSEa621vx7kfSV030WrzUv7rpu1thbYQGh8d6h9xrPugWq7xFp7xIa0Aj/j1PfdJ/lcFwNXGGP2Ebob6MWEjrA9td8ixDI+EsuD0NjNHkKD6t0D6HPi1X+fbWUC2WHtTYS+Zf1XIr9E+BenfRmRX1S858wfQ+gb6FznsRcY4yz7s7Nu9xcVy2Koq4DIcWDX6+lvGzHUlh/WvgV4xmnPIfILkj2Evhzp9/MFniPyC5LrnfY3ifwS5tkodRngF8D9feYnfd8NUJtX9t04YLTTTgfeApYPtc941h1Dbflh+/Z+4N5k/U04y5fS+2Vi0vdbv3XGOUCXEfpmfDdwezz77rOd6c6b7z7153Znfh6hLwh2Os/dH6gh9OMHuwmdzlMc1tfXCZ1CswtYFTa/GChzXvMfDP4l2NOE/je4ndB/Uf82EfX0t40YanvS2fYWQvduCQ+f253t7CDsbJf+Pl/n83jPqfk5INWZn+ZM73KWT49S2wWE/vdvC2Gnu3lh3w1Qm1f23VxCp5dtcd7fHafaZ7zqjqG23zv7rgx4it4zQxL6NxHWx1J6gzrp+62/hy4hFxHxOF2ZKCLicQpqERGPU1CLiHicglpExOMU1CIiHqegFhHxOAW1iIjH/X/tNnV1a/D0WwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.31014543771743774.\n",
      "Better model found at epoch 1 with accuracy value: 0.3261563181877136.\n",
      "Better model found at epoch 2 with accuracy value: 0.3323418200016022.\n",
      "Better model found at epoch 3 with accuracy value: 0.34127548336982727.\n",
      "Better model found at epoch 4 with accuracy value: 0.3503599464893341.\n",
      "Better model found at epoch 5 with accuracy value: 0.3607962131500244.\n",
      "Better model found at epoch 6 with accuracy value: 0.3721919059753418.\n",
      "Better model found at epoch 7 with accuracy value: 0.3838889002799988.\n",
      "Better model found at epoch 8 with accuracy value: 0.393983393907547.\n",
      "Better model found at epoch 9 with accuracy value: 0.3968702554702759.\n",
      "CPU times: user 5h 51min 50s, sys: 1h 41min 8s, total: 7h 32min 59s\n",
      "Wall time: 7h 31min 56s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "learn.unfreeze()\n",
    "wd = 0.01 \n",
    "learn.fit_one_cycle(10, lr, wd=wd, moms=(0.8,0.7), \n",
    "                    callbacks=[ShowGraph(learn),\n",
    "                               SaveModelCallback(learn.to_fp32(), monitor='accuracy', name='bestmodel_sp15_multifit')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Save the pretrained model and vocab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "learn.to_fp32().save(mdl_path/lm_fns3[0], with_opt=False)\n",
    "learn.data.vocab.save(mdl_path/(lm_fns3[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Backward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15_multifit_bwd', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 952 ms, sys: 68 ms, total: 1.02 s\n",
      "Wall time: 1.02 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, drop_mult=0., pretrained=False, \n",
    "                               metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxU9Znv8c/TGzRb00CDDQ00KCDGhUiLEhMUtxg1UZPo1SR3HHUumUxMNBmzzHiz3PjKZDGJM1lmHJMYHdfEqBMTJy5xAZOA0qBgCyiCNDRbN73R9L4894+qlqLthgbq1DnV9X2/XvXqql+dOr+HovrpX/3O7zzH3B0REckcWWEHICIiqaXELyKSYZT4RUQyjBK/iEiGUeIXEckwOWEHMBgTJkzw0tLSsMMQEUkrq1at2uPuRX3b0yLxl5aWUl5eHnYYIiJpxcwq+2sPbKrHzO4ys2ozq0ho+6aZbTezV+O3i4LqX0RE+hfkHP/dwIX9tN/u7vPit/8JsH8REelHYInf3ZcBdUHtX0REjkwYq3puMLO18amgwoE2MrMlZlZuZuU1NTWpjE9EZEhLdeL/D+BYYB6wE/jhQBu6+53uXubuZUVF7zooLSIiRyilid/dd7t7t7v3AD8HFqSyfxERSXHiN7PihIeXAxUDbSsiIsEIbB2/mT0InA1MMLMq4BvA2WY2D3BgC/DpoPoXEUlnuxrbuP+lSi5/7xRmFo1K6r4DS/zufnU/zb8Mqj8RkaGksraZnzz3FmfMHJ/0xK9aPSIiEdTQ2glAQX5u0vetxC8iEkENLR0AjB2hxC8ikhEaWmIj/sIReUnftxK/iEgENbR2kpttjMjLTvq+lfhFRCKooaWDgvw8zCzp+1biFxGJoIaWTgoDmN8HJX4RkUhqaOkM5MAuKPGLiERSQ2snBfnJP7ALSvwiIpHU0NKhEb+ISCbRHL+ISAZp6+ymtbObsQGs4QclfhGRyGkMsFwDKPGLiEROkGftghK/iEjkBFmnB5T4RUQip75FUz0iIhmlsTU24i8cqakeEZGM0DvHP1YjfhGRzFDfElxlTlDiFxGJnMbW4CpzghK/iEjkBHnWLijxi4hETpCVOUGJX0QkcurjF2EJSmCJ38zuMrNqM6vo57mbzczNbEJQ/YuIpKvG1vSd6rkbuLBvo5lNBc4HtgbYt4hI2krbqR53XwbU9fPU7cCXAQ+qbxGRdBV0ZU5I8Ry/mX0E2O7uawax7RIzKzez8pqamhREJyISvt7KnGk54u/LzEYAtwBfH8z27n6nu5e5e1lRUVGwwYmIRMT+s3aHxoj/WGAGsMbMtgAlwGozOyaFMYiIRFp9wJU5AXIC23Mf7v4aMLH3cTz5l7n7nlTFICISdQ0BV+aEYJdzPggsB+aYWZWZXR9UXyIiQ0XQlTkhwBG/u199iOdLg+pbRCRdBV2ZE3TmrohIpARdmROU+EVEIqWxtYOxI4KrzAlK/CIikdLQ0hnoNA8o8YuIREp9S0egSzlBiV9EJFJidXqCW9EDSvwiIpHS2KqpHhGRjKKpHhGRDNLW2U1bZ4+mekREMkUqKnOCEr+ISGS8U6AtwMqcoMQvIhIZ75Rr0IhfRCQzKPGLiGSY3sqcOrgrIpIh6lNQmROU+EVEIqOhpZO87KxAK3OCEr+ISGQ0tnZQMCI30MqcoMQvIhIZ9c3Bl2sAJX4RkchoaA2+XAMo8YuIREZDSycFAZ+8BUr8IiKRUdvcwYRRSvwiIhmhu8ep3dfOhFHDAu9LiV9EJALqWzrocSgancaJ38zuMrNqM6tIaLvVzNaa2atm9rSZTQ6qfxGRdFLT1A6Q9iP+u4EL+7Td5u4nu/s84A/A1wPsX0QkbezZF0v8aT3id/dlQF2ftr0JD0cCHlT/IiLppHfEn4rEnxN4D32Y2beBvwEagcUH2W4JsARg2rRpqQlORCQkvSP+Ibmqx91vcfepwP3ADQfZ7k53L3P3sqKiotQFKCISgpqmdobnZjFqWPDj8TBX9TwAfCzE/kVEIqOmKbaUM+g6PZDixG9msxIefgTYkMr+RUSias++jpTM70OAc/xm9iBwNjDBzKqAbwAXmdkcoAeoBP4+qP5FRNJJTVM708aPSElfgSV+d7+6n+ZfBtWfiEg627OvnfmlhSnpS2fuioiErKu7h7qWDopScPIWKPGLiISurrkDd5iQojl+JX4RkZBV9568pRG/iEhmqHmnXEPwJ2+BEr+ISOj2vDPiH56S/pT4RURC1jvin6ARv4hIZtjT1MHIvGxG5KWmfJoSv4hIyGr2tafsrF1Q4hcRCd2eptRccrGXEr+ISMg04hcRyTB7UnSR9V5K/CIiIWrv6qahpVMjfhGRTFG7rwNIzUXWeynxi4iEKJUXWe+lxC8iEqLei6yn4lq7vZT4RURCpBG/iEiG2T/iV+IXEckIe/Z1MHp4DsNzs1PWpxK/iEiIappSe/IWKPGLiISqJsUnb4ESv4hIqPZoxC8ikllqmtpTdsnFXoElfjO7y8yqzawioe02M9tgZmvN7DEzGxtU/yIiUdfW2U1Te9eQGvHfDVzYp+0Z4ER3Pxl4E/inAPsXEYm0mhRfZL1XYInf3ZcBdX3annb3rvjDFUBJUP2LiETdnhRfcrFXmHP81wF/HOhJM1tiZuVmVl5TU5PCsEREUqMmxRdZ7xVK4jezW4Au4P6BtnH3O929zN3LioqKUheciEiK7OmtzJniEX9qruybwMyuAS4BznV3T3X/IiJRsWtvG2YwfmQE5/jN7FgzGxa/f7aZff5IVuSY2YXAV4CPuHvL4b5eRGQo2V7fyjFjhpOXk9rJl8H29gjQbWbHAb8EZgAPHOwFZvYgsByYY2ZVZnY98FNgNPCMmb1qZncceegiIumtqr6FksL8lPc72KmeHnfvMrPLgX9195+Y2SsHe4G7X91P8y8PO0IRkSGqqr6VBTPGpbzfwY74O83sauAa4A/xttxgQhIRGfq6unvYtbctlBH/YBP/tcBC4Nvu/raZzQDuCy4sEZGhbWdjG909Ht2pHndfB3wewMwKgdHu/t0gAxMRGcqq6lsBKCkckfK+B7uq5wUzG2Nm44A1wK/M7EfBhiYiMnRV1ccWNk4ZG92pngJ33wt8FPiVu88HzgsuLBGRoa2qvhUzKB6b2rN2YfCJP8fMioEr2X9wV0REjlBVfSuTRg9nWE7qLrnYa7CJ/1vAU8Amd19pZjOBjcGFJSIytG1vCGcNPwz+4O7DwMMJjzcDHwsqKBGRoa6qvpWy6YWh9D3Yg7sl8QunVJvZbjN7xMxUUllE5Ah0dfews7EtlBU9MPipnl8BjwOTgSnA7+NtIiJymHbtDW8NPww+8Re5+6/cvSt+uxtQrWQRkSMQ5hp+GHzi32NmnzKz7PjtU0BtkIGJiAxV+xN/tEf81xFbyrkL2Al8nFgZBxEROUxV9S2hreGHQSZ+d9/q7h9x9yJ3n+julxE7mUtERA5TVX0rE0cPC2UNPxzdpRe/mLQoREQySKwOfzjz+3B0id+SFoWISAapqm8NbX4fji7x63q5IiKHaf8a/vAS/0HP3DWzJvpP8AaEF7WISJra3dQeX8Mf3lTPQRO/u49OVSAiIpmgqi5Wjjldp3pEROQwhX3yFijxi4ikVG/inxzSGn5Q4hcRSamq+hYmjQlvDT8EmPjN7K54Nc+KhLYrzOx1M+sxs7Kg+hYRiarYUs7wpnkg2BH/3cCFfdoqiJ3xuyzAfkVEIquqoSWU6+wmCizxu/syoK5P23p3fyOoPkVEoqyru4edDeGu4QfN8YuIpMyuvW10hbyGHyKc+M1siZmVm1l5TU1N2OGIiBy1rbWxNfyl45X4++Xud7p7mbuXFRXpmi8ikv62xBP/NCV+EZHMUFnbTF52FsUFQ3SO38weBJYDc8ysysyuN7PLzawKWAg8YWZPBdW/iEjUVNa2MHVcPtlZ4RY3PmitnqPh7lcP8NRjQfUpIhJlW2qbmT5+ZNhhaKpHRCQV3J2tdS1MD3l+H5T4RURSomZfOy0d3ZRqxC8ikhm2RmRFDyjxi4ikxJZ31vBrxC8ikhEqa5vJMkKv0wNK/CIiKVFZ28KUwnzycsJPu+FHICKSASprmyMxzQNK/CIiKVFZ18K0ceEf2AUlfhGRwDW0dNDQ0qkRv4hIpqiM0FJOUOIXEQlcZV10lnKCEr+ISOAq9zQDaI5fRCRTVNa1MGnMMPLzssMOBVDiFxEJXGVEqnL2UuIXEQnYltqW0C+3mEiJX0QkQC0dXdQ0tWvELyKSKXqXckahDn8vJX4RkQC9k/jHacQvIpIRKmvjSzk14hcRyQyVdS2MG5lHQX5u2KG8Q4lfRCRAm2v2RWp+H5T4RUQC4+6s39nE8ceMCTuUAyjxi4gEZGdjG42tnZxQPDrsUA4QWOI3s7vMrNrMKhLaxpnZM2a2Mf6zMKj+RUTCtn7nXgDmFmfOiP9u4MI+bV8FnnX3WcCz8cciIkNSb+I/PlMSv7svA+r6NF8K3BO/fw9wWVD9i4iEbf3OJqaNG8GoYTlhh3KAVM/xT3L3nQDxnxMH2tDMlphZuZmV19TUpCxAEZFkWb9zL3MjNr8PET646+53unuZu5cVFRWFHY6IyGFp6eji7drmyM3vQ+oT/24zKwaI/6xOcf8iIinxxq4m3KN3YBdSn/gfB66J378G+F2K+xcRSYn1O5sAOCGTEr+ZPQgsB+aYWZWZXQ98FzjfzDYC58cfi4gMOet37mX0sBxKCvPDDuVdAjvU7O5XD/DUuUH1KSISFet37uX44tGYWdihvEtkD+6KiKSrnh5nw66mSM7vgxK/iEjSVdW3sq+9S4lfRCRTrItoqYZeSvwiIkm2fudesgzmTIreyVsQ4MHdKNjX3kV7Z3cg+w7qgM3h7PVgIdih9pTwdN/92Dvt795L77a9z1if/SS2W8I+zIJ7z0SiZv3OvZROGEl+XnbYofRrSCf+7/1xA/euqAw7DEmQ+AchK/6Hwgyy4o+zLPY4O8tibVlGTvx+TraRnWXkZWeRk23kZmcxLCeL4bnZDM/JJj8vmzHDcxiTn8uY4bkUjR5GSWE+JYUjmDh6GFlZ+sMjqbF+115OLhkbdhgDGtKJ/+KTi5k1aVTS9+ue9F3G9zv4HR9sy0PtJvHpgfp0B+/TS++m3s82ibtx9/hzsfYej28V/9n7uthz8e2JrYTodn/nNd09/s7Prp79P7u6e+jsdjq7e2jv7KGuuYO2zm5aOrrZ29pJU3vXu96DvJwsTikpYMGMcZw+YzzzpxcyMmKFs2RoaGrrZFtdK1edNi3sUAY0pD/5Z8wczxkzx4cdhqRYT4/T1N5FTVMb2+pbqapvZcueZsor67lj6WZ+9vwm8nKyOGt2EZecXMy5cydFrnqipK8Nu2Jn7EaxOFsvfdplyMnKMgrycynIz+W4iQf+8u1r72J1ZT3Pv1HN/7y2k2fW7SYvJ4vzT5jElWVTef9xE8jWlJAchdWV9QCcOKUg5EgGpsQvGWXUsBwWzS5i0ewivnbxCazaWs/v1+zg8TU7eGLtTooLhvPx+SV86ozpTBozPOxwJQ2t2FzLsUUjmTg6up8fJX7JWFlZxmml4zitdBy3XDyXZ9dX85vybfzs+be4Y+kmLps3hSWLZjIrokvyJHq6untYuaWeS+dNDjuUg1LiFwGG5WRz0UnFXHRSMVtrW/jFnzfzm/JtPLyqinOPn8hnzj6WstJxYYcpEVexYy/72rtYeGy0jy3qBC6RPqaNH8G3Lj2Rv371XL5w3mxWb63n43cs54o7/spzG3Yf1uorySzLN9UCRH5RiRK/yADGjczjxvNm8ZevnsM3PnwC2+tbue7ucj70by/yu1e309XdE3aIEjErNtcya+IoJowaFnYoB6XEL3III/JyuPbMGSz98mJ+cMUpdPU4Nz70Kuf8cCn3v1RJe1cwZ4dLeuns7mHllrrIT/OAEr/IoOVmZ/Hx+SU8fdMi/vN/z6dwZB63PFbBOT9YygMvbaWjS98AMtnaqkZaOrojP80DSvwihy0ry/jge47hv//hfdxz3QKKRg/jnx97jXN++AK/XrmVzpCmgHp6nHtXVFK+pU7HIUKwYnNsfv/0GdFfBKBVPSJHyMw4a3YRi2ZN4IU3a7j9mTf5yiOv8bPnN/H5c2dx2bzJ5GSnbmy1fHMtX/vvCgCmjx/BZfOm8NFTpzB9/MiUxZDJVmyuZc6k0YyP+Pw+aMQvctTMjMVzJvK7z57JL68pY/TwHG5+eA3n376MJ9buTNnoe/mmWrKzjO989CRKCvP58XMbOeu2F/jEz1fw+JodOhYRoI6uHsq31KfF/D5oxC+SNGbGuXMncc7xE3lm3W5+9MybfPaB1bx32lj+78VzmT892CmA5ZtrOWlKAVcvmMbVC6axo6GVR1dX8dDKbXz+wVcoHJHLVQumce2ZpZE+qzQdra1qoLWzmzNmRn+aBzTiF0k6M+OC9xzDE5//AN//2Mlsr2/lY/+xnM/ct4pNNfsC6bO5vYs12xoOGHFOHpvPDefMYtmXFnPv9Qs4fcZ47li6ifd/73n++bHX2LKnOZBYMtHyTbWYwekzNOIXyWjZWcaVp03lklOKuXPZZn6+bDNPr9vNFfNLuPG8WRQX5Cetr/LKerp6nIX9rCjJyjI+MKuID8wq4u09zdy5bDO/La/iwZe3snjORD6xYBqLj5+o4nRHYcXbtRx/zBgKR+aFHcqghDLiN7MbzazCzF43s5vCiEEkVUbk5XDTebNZ+uXF/M3C6Ty6ejtn3fYCt/5hHTVN7UnpY/mmWnKzjbLSwoNuN2PCSL7z0ZP481cWc8Pi46jY3sjf/Vc57//ec/zk2Y3UNXckJZ5MsntvGyu31PO+NJnfB7BUL/sysxOBh4AFQAfwJPAZd9840GvKysq8vLw8RRGKBKuqvoV//dNGHl1dRV5OFtcsLGXJoplHtRrk0p/9hdws47efed9hva6zu4dn11dz/0uVvLhxD8Nzs7hi/lSuf/8MSidoNdBgfO2/K3jw5a08949nM238iLDDOYCZrXL3sr7tYYz45wIr3L3F3buApcDlIcQhEoqSwhH84IpTePYfz+aiE4v5+Yub+cD3n+fbT6yjem/bYe+vqa2Tiu2NR7SiJDc7iwtPPIZ7rz+dp7+wiI+cMplfr9zG4h++wPV3r+T5N6rp6dE5AQPZVtfCQyu3cuVpUyOX9A8mjBH/XOB3wEKgFXgWKHf3z/XZbgmwBGDatGnzKyt17VwZmt6q3sdPn9vI42t2kJOdxZVlJXx60bFMHTe4RPLcht1cd3c5D/zd6bzvuAlHHU/13jb+a3klD63cxp597Uwdl88nT5/O/yqbmjZz2Kly88NreHzNDpZ+6eykHrNJloFG/ClP/PFgrgc+C+wD1gGt7v6FgbbXVI9kgsraZu5YuplHVlXR7c7FJxXz6bNm8p7JB7+S07efWMc9f61k7TcvYHhudtLi6ejq4anXd3HfikpeeruOYTlZXDpvMte8r/SQMWWCt6r3ccHtS7n2zBl87ZITwg6nX5FK/AcEYPYvQJW7//tA2yjxSybZ1djGXX95m/tXVNLc0c0HZk3ghsXHcfoANWAu+cmLjMzL4defXhhYTG/sauKe5Vt4bPV2Wju7Oa20kE+dMZ0PnVhMXk5mrgr/7AOreX5DNcu+vDiy1TgjlfjNbKK7V5vZNOBpYKG71w+0vRK/ZKLG1k7uf6mSu/68hT372jlj5jhuPHf2AXP5jS2dzLv1aW48dxY3nTc7+JhaOvlN+Tbue6mSytoWJozK46rTpnHVgqmUFKbPHPfRen1HIxf/+M/csPg4bv7gnLDDGVDUEv+LwHigE/iiuz97sO2V+CWTtXV288BLW7lj6Saqm9pZUDqOa88s5bwTJvH8hmqW3LuKXy85Y8BvBEHo6XGWbazhvhWVPLuhGoDFcybyydOncfacoX1OQHePc9Wdy3ljVxMvfuUcCvJzww5pQJFK/IdLiV8k9gfgwZe38osX32Z7QysTRw9j4phhbNy9j7XfvIBhOcmb3z8cVfUtPPTyNn5dvo2apnYmFwznirKpXFFWMiS/Bfzs+be47ak3+NGVp/DRU0vCDueglPhFhojuHueFN6q5d0UlS9+s4azZRdx97YKww6Kzu4c/rdvNgyu38eLGGgAWzSriirISzps7KakHnsNSsb2Ry372Fz74nmP46Sfei1m0v9ko8YsMQTsaWhmZl0PBiGhNN2yra+HhVVU8XL6NnY1tjB6ewyUnF3P5e0s4rbQw8gmzP60d3VzykxfZ197FUzctYuyI6C9tVeIXkZTr7nGWb6rl0VeqeLJiFy0d3UwfP4KPnVrCR0+dklZTQd/4XQX3LK/kvutP5/2zjv58iVRQ4heRUDW3d/FkxS4eWV3FXzfFrlZ1xsxxfPiUyXzoxGLGRfTkMHfnjqWb+d6TG7j2zFK+8eH3hB3SoCnxi0hkbKtr4bFXtvO7V7ezqaaZ7CzjzOMmcPFJx3De3EmRuYpVW2c3//zoazz6ynY+fMpkbvv4yWl1rEKJX0Qix91Zv7OJP6zdwe/X7mBbXStZ8br2H3zPJM6dO2nQpSuSrbqpjU/fu4pXtjbwxfNn87lzjku7YxNK/CISae7Oup17ebJiF09W7GJjdeyiNbMmjuKc4ydy1uwiTp1eGPiIu7vHebh8G7c99QYtHd386MpT+NBJxYH2GRQlfhFJK2/vaea5DdU8t2E3L22uo6vHycvOYt60sZwxYxwnl4xl7uQxTC4YnrSR+MotdXzz8dd5fcdeTist5NbLTuT4Y8YkZd9hUOIXkbTV1NbJyi11rNhcx4rNtVRsb6S3WnRBfi5zjhnNsUUjmTlhFDMmjOSYguGMG5nHuJF5A35D6O5x6ls6WFVZz/JNtazYXMuGXU0UFwznny6ay4dPLk67qZ2+lPhFZMhobu9iw64m1u3cy7ode3lzdxNv72nu9wpiw3KyyM/LZnhONsNys+jucRpbO9nX3kVv+huWk0VZaSFnzS7iU2dMZ0Te0Lgq7UCJf2j860Qko4wclsP86YXMn37gpSYbWjrYvKeZmqZ26ps7qGvpoLGlk9bObto7e2jr6ibLjIL8XMbk5zI2P5cTpxRwytSC0EpehEGJX0SGjLEj8jh1WjTPB4iSzCykLSKSwZT4RUQyjBK/iEiGUeIXEckwSvwiIhlGiV9EJMMo8YuIZBglfhGRDJMWJRvMrAZoABr7PFVwiLZD3e/9OQHYcwSh9df/4cR3qJj7izXx+SOJ+0hiPlhc/T3uL9ajea9TGXPi/ah/PqISc3/t+nwcWio+H2Pdvehde3b3tLgBdx5u26HuJ/wsT1ZMyYx5gFgTtz3suI8k5oPFNZj392jf61TGnE6fj6jErM9H9D8ffW/pNNXz+yNoO9T9/l5/tDEd6vnDiTnxcZgx99d+sMf9xXo0cacy5sT7Uf98RCXm/tr1+Ti0VH4+DpAWUz1BM7Ny76eCXdSlY9yKOTXSMWZIz7jTMeZ0GvEH6c6wAzhC6Ri3Yk6NdIwZ0jPutItZI34RkQyjEb+ISIZR4hcRyTBDLvGb2V1mVm1mFUfw2vlm9pqZvWVmP7aEC26a2efM7A0ze93Mvh/1mM3sm2a23cxejd8uSmbMQcWd8PzNZuZmNiF5EQf2Xt9qZmvj7/PTZjY5DWK+zcw2xON+zMzGpkHMV8R//3rMLGkHU48m1gH2d42ZbYzfrkloP+hnPqWOZP1plG/AIuBUoOIIXvsysBAw4I/Ah+Lti4E/AcPijyemQczfBG5Ot/c6/txU4CmgEpgQ9ZiBMQnbfB64Iw1ivgDIid//HvC9NIh5LjAHeAEoCzvWeBylfdrGAZvjPwvj9wsP9u8K4zbkRvzuvgyoS2wzs2PN7EkzW2VmL5rZ8X1fZ2bFxH6Bl3vsf+m/gMviT38G+K67t8f7qE6DmAMXYNy3A18Gkr7yIIiY3X1vwqYjkx13QDE/7e5d8U1XACVpEPN6d38jmXEeTawD+CDwjLvXuXs98AxwYdi/q30NucQ/gDuBz7n7fOBm4N/72WYKUJXwuCreBjAb+ICZvWRmS83stECjjTnamAFuiH+Vv8vMDrwqdXCOKm4z+wiw3d3XBB1ogqN+r83s22a2Dfgk8PUAY+2VjM9Hr+uIjUCDlsyYgzaYWPszBdiW8Lg3/qj8u4AMuNi6mY0C3gc8nDClNqy/Tftp6x255RD72nYGcBrwGzObGf/LnXRJivk/gFvjj28FfkjsFzwwRxu3mY0AbiE2DZESSXqvcfdbgFvM7J+AG4BvJDnU/YEkKeb4vm4BuoD7kxnjuwJJYsxBO1isZnYtcGO87Tjgf8ysA3jb3S9n4PhD/3clGvKJn9i3mgZ3n5fYaGbZwKr4w8eJJcrEr7slwI74/Srg0Xiif9nMeogVZqqJaszuvjvhdT8H/hBQrImONu5jgRnAmvgvXAmw2swWuPuuiMbc1wPAEwSY+ElSzPEDj5cA5wY1iEmQ7Pc5SP3GCuDuvwJ+BWBmLwB/6+5bEjapAs5OeFxC7FhAFeH/u/YL6+BCkDeglIQDNcBfgSvi9w04ZYDXrSQ2qu89+HJRvP3vgW/F788m9lXOIh5zccI2XwAeSof3us82W0jywd2A3utZCdt8DvhtGsR8IbAOKAricxHkZ4MkH9w90lgZ+ODu28RmCArj98cN9jOfqlsonQb6D4IHgZ1AJ7G/stcTG0U+CayJf9i/PsBry4AKYBPwU/af2ZwH3Bd/bjVwThrEfC/wGrCW2EiqOJkxBxV3n222kPxVPUG814/E29cSK4w1JQ1ifovYAObV+C3ZK5GCiPny+L7agd3AU2HGSj+JP95+Xfz9fQu49nA+86m6qWSDiEiGyZRVPSIiEqfELyKSYZT4RUQyjBK/iEiGUeIXEckwSvySlsxsX4r7+4WZnZCkfXVbrJJnhZn9/lCVMc1srJn9QzL6FgFdgUvSlJntc/dRSdxfju8vWhaoxNjN7B7gTXf/9kG2LwX+4O4npiI+Gfo04pchw8yKzOwRM1sZv50Zb19gZn81s1fiP+fE2//WzB42s98DT5vZ2Wb2gpn91mK16u/vrZwUCacAAAJ4SURBVJkeby+L398XL8q2xsxWmNmkePux8ccrzexbg/xWspz9BepGmdmzZrbaYnXbL41v813g2Pi3hNvi234p3s9aM/t/SXwbJQMo8ctQ8m/A7e5+GvAx4Bfx9g3AInd/L7HKmf+S8JqFwDXufk788XuBm4ATgJnAmf30MxJY4e6nAMuA/5PQ/7/F+z9kHZZ4nZpziZ1ZDdAGXO7upxK7BsQP4394vgpscvd57v4lM7sAmAUsAOYB881s0aH6E+mVCUXaJHOcB5yQUFFxjJmNBgqAe8xsFrGKiLkJr3nG3RNrsb/s7lUAZvYqsRouf+7TTwf7i96tAs6P31/I/hrrDwA/GCDO/IR9ryJWsx1iNVz+JZ7Ee4h9E5jUz+sviN9eiT8eRewPwbIB+hM5gBK/DCVZwEJ3b01sNLOfAM+7++Xx+fIXEp5u7rOP9oT73fT/O9Lp+w+ODbTNwbS6+zwzKyD2B+SzwI+J1fIvAua7e6eZbQGG9/N6A77j7v95mP2KAJrqkaHlaWK18AEws96yugXA9vj9vw2w/xXEppgArjrUxu7eSOxSjTebWS6xOKvjSX8xMD2+aRMwOuGlTwHXxevGY2ZTzGxikv4NkgGU+CVdjTCzqoTbF4kl0bL4Ac91xMppA3wf+I6Z/QXIDjCmm4AvmtnLQDHQeKgXuPsrxCpAXkXsYihlZlZObPS/Ib5NLfCX+PLP29z9aWJTScvN7DXgtxz4h0HkoLScUyRJ4lcQa3V3N7OrgKvd/dJDvU4k1TTHL5I884GfxlfiNBDwpS5FjpRG/CIiGUZz/CIiGUaJX0Qkwyjxi4hkGCV+EZEMo8QvIpJh/j8c1U/tZ3DSdAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 3e-3\n",
    "lr *= bs/48  # Scale learning rate by batch size\n",
    "\n",
    "wd = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.759265</td>\n",
       "      <td>3.812161</td>\n",
       "      <td>0.645454</td>\n",
       "      <td>0.354546</td>\n",
       "      <td>45.248158</td>\n",
       "      <td>45:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.524330</td>\n",
       "      <td>3.643459</td>\n",
       "      <td>0.632401</td>\n",
       "      <td>0.367599</td>\n",
       "      <td>38.223717</td>\n",
       "      <td>45:16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.399805</td>\n",
       "      <td>3.569191</td>\n",
       "      <td>0.624596</td>\n",
       "      <td>0.375404</td>\n",
       "      <td>35.487923</td>\n",
       "      <td>45:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.358356</td>\n",
       "      <td>3.492392</td>\n",
       "      <td>0.615690</td>\n",
       "      <td>0.384309</td>\n",
       "      <td>32.864567</td>\n",
       "      <td>48:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.298991</td>\n",
       "      <td>3.423204</td>\n",
       "      <td>0.607334</td>\n",
       "      <td>0.392666</td>\n",
       "      <td>30.667429</td>\n",
       "      <td>44:30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.230762</td>\n",
       "      <td>3.344130</td>\n",
       "      <td>0.598109</td>\n",
       "      <td>0.401891</td>\n",
       "      <td>28.335854</td>\n",
       "      <td>44:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.139510</td>\n",
       "      <td>3.259253</td>\n",
       "      <td>0.586869</td>\n",
       "      <td>0.413130</td>\n",
       "      <td>26.030067</td>\n",
       "      <td>44:16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.123008</td>\n",
       "      <td>3.175821</td>\n",
       "      <td>0.575663</td>\n",
       "      <td>0.424338</td>\n",
       "      <td>23.946394</td>\n",
       "      <td>44:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.007710</td>\n",
       "      <td>3.114435</td>\n",
       "      <td>0.566360</td>\n",
       "      <td>0.433641</td>\n",
       "      <td>22.520712</td>\n",
       "      <td>44:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.955673</td>\n",
       "      <td>3.098413</td>\n",
       "      <td>0.563313</td>\n",
       "      <td>0.436688</td>\n",
       "      <td>22.162729</td>\n",
       "      <td>44:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdeklEQVR4nO3deXRc5Z3m8e+vSqXSUpJsyZu8IdkQbBbFGONxgAE3WQCzJGk8xJkkkyaZ8UlIT4DuTGI60wnpYTqQziRMTjcGkibpTGg2Q042IAlpHDNjApGNMcJLvOBFXmXZ2reS9M4fdSVXySWpZNeVrq3nc45O3fve7Ve3rEfX713KnHOIiEhwhca6ABERGZqCWkQk4BTUIiIBp6AWEQk4BbWISMDl+LHSkoml7oK5c/xYtYjIOWnDhg3HnHOT003zJainTJ9FdXW1H6sWETknmdnewaap60NEJOAU1CIiAaegFhEJOF/6qEVERiIej1NbW0tHR8dYl+K7vLw8Zs6cSSQSyXgZX4JaTw8RkZGora2lqKiIiooKzGysy/GNc476+npqa2uprKzMeDl/uj6U1CIyAh0dHZSVlZ3TIQ1gZpSVlY34fw7qoxaRQDjXQ7rP6bxPBbWISMApqEVk3GtoaODhhx8e8XLLli2joaHBh4pSKahFZNwbLKh7enqGXO6FF15gwoQJfpXVz6fL83Q2UUTOHqtWrWLXrl0sWLCASCRCLBajvLycTZs2sWXLFj7ykY+wf/9+Ojo6uOuuu1i5ciUAFRUVVFdX09LSwo033sjVV1/N+vXrmTFjBj/72c/Iz8/PSn26jlpEAuUbv3iHLQebsrrOi6YX8/VbLh50+gMPPEBNTQ2bNm1i7dq13HTTTdTU1PRfQvf4449TWlpKe3s7V1xxBbfddhtlZWUp69ixYwdPPvkk3//+97n99tt57rnn+OQnP5mV+hXUIiIDLF68OOU65+9973v89Kc/BWD//v3s2LHjlKCurKxkwYIFAFx++eXs2bMna/XohhcRCZShjnxHS2FhYf/w2rVrefnll3nttdcoKChg6dKlaa+Djkaj/cPhcJj29vas1aOTiSIy7hUVFdHc3Jx2WmNjIxMnTqSgoIBt27bxhz/8YZSrU9eHiAhlZWVcddVVXHLJJeTn5zN16tT+aTfccAOPPPIIVVVVXHjhhSxZsmTU6zPnst9RMfeiKrdry+asr1dEzk1bt25l/vz5Y13GqEn3fs1sg3NuUbr51fUhIhJwCmoRkYDT0/NERAJOR9QiIgHnS1DrgFpEJHt0RC0iEnAKahGR0xCLxQA4ePAgy5cvTzvP0qVLqa6uPuNtKahFRM7A9OnTWbNmja/b0J2JIiLAV77yFc477zzuvPNOAO677z7MjHXr1nHixAni8Tj3338/H/7wh1OW27NnDzfffDM1NTW0t7dzxx13sGXLFubPn5+1530oqEUkWF5cBYffzu46p10KNz4w5CwrVqzg7rvv7g/qZ555hpdeeol77rmH4uJijh07xpIlS7j11lsH/d7D1atXU1BQwObNm9m8eTMLFy7MSvkKahER4LLLLuPo0aMcPHiQuro6Jk6cSHl5Offccw/r1q0jFApx4MABjhw5wrRp09KuY926dXzxi18EoKqqiqqqqqzUpqAWkWAZ5sjXT8uXL2fNmjUcPnyYFStW8MQTT1BXV8eGDRuIRCJUVFSkfcRpMj++TV3XUYuIeFasWMFTTz3FmjVrWL58OY2NjUyZMoVIJMIrr7zC3r17h1z+mmuu4YknngCgpqaGzZuz83A6HVGLiHguvvhimpubmTFjBuXl5XziE5/glltuYdGiRSxYsIB58+YNufznP/957rjjDqqqqliwYAGLFy/OSl2+POa0cn6Ve3erHnMqIpnRY06z8JhTM7vHzN4xsxoze9LM8rJQq4iIZGDYoDazGcAXgUXOuUuAMLBiyIXUSS0ikjWZnkzMAfLNLAcoAA76V5KIjEd+dMMG0em8z2GD2jl3APg2sA84BDQ6534zcD4zW2lm1WZW3dLSMuJCRGT8ysvLo76+/pwPa+cc9fX15OWNrPd42JOJZjYReA74GNAAPAuscc79ZLBlKuZd6vZsy/KdRSJyzorH49TW1g57jfK5IC8vj5kzZxKJRFLahzqZmMnleR8A3nXO1Xkrex64Ehg0qEVERiISiVBZWTnWZQRWJn3U+4AlZlZgiVtu3g9s9bcsERHpk0kf9evAGmAj8La3zGM+1yUiIp6M7kx0zn0d+LrPtYiISBr64gARkYBTUIuIBJyCWkQk4BTUIiIBp+dRi4gEnD9H1EpqEZGsUdeHiEjAKahFRAJOQS0iEnAKahGRgFNQi4gEnC7PExEJOB1Ri4gEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJwenqeiEjA+XTDi5JaRCRb/Alq5bSISNb4EtQ9SmoRkazxJah7FdQiIlnjT1D3+rFWEZHxSUfUIiIB508fda+CWkQkW3RELSIScD4FtR9rFREZn3w6maikFhHJFnV9iIgEnG/fmdjVrWv0RESywben57V0dvu1ahGRccW3oG7uiPu1ahGRccW3oI73qJ9aRCQbfAvqti51fYiIZENGQW1mE8xsjZltM7OtZva+4ZZp6VBQi4hkQ6ZH1P8beMk5Nw94L7B1uAVW/37XmdQlIiKenOFmMLNi4BrgLwCcc11A13DLvbrj2JnWJiIiZHZEPQeoA35oZm+a2Q/MrHDgTGa20syqzaw661WKiIxjmQR1DrAQWO2cuwxoBVYNnMk595hzbpFzblGWaxQRGdcyCepaoNY597o3voZEcIuIyCgYNqidc4eB/WZ2odf0fmCLr1WJiEi/YU8mev4r8ISZ5QK7gTv8K0lERJJlFNTOuU3AiPuenXOY2YiLEhGRk3y7MxGgsV3P+xAROVO+BvWWQ01+rl5EZFzwJagnxaIAfOPnOucoInKmfAnqorxE1/f2I81+rF5EZFzxJagLczO9mERERIbjS1DrQg8Rkezx9WQiwJGmDr83ISJyTvM9qA83KqhFRM6Eb0F9wZQYAB/+p//n1yZERMYF34L6gduq/Fq1iMi44ltQX37exP5hdX+IiJw+3/uoAZZ883ejsRkRkXOSr0F959K5fq5eRGRc8DWo/9v1F/YPd/f0+rkpEZFzlq9BnfyI0/O/+qKfmxIROWf53kddkh/xexMiIuc034P6zb/9YP/wlTqpKCIyYr4HdSh0svvjoC7TExEZsVG5PG/PAzf1D69eu2s0Nikics4YlaBO9uBL20Z7kyIiZ7VRC+qX7v73/cNf+1nNaG1WROSsN2pBPW9acf/wj1/bS3OHvvhWRCQTo9r18dq91/UPX3rfb0Zz0yIiZ61RDeryknxuee/0/vE9x1pHc/MiImelUT+Z+A/LTz7+dOm319IR7xntEkREziqjHtR5kTBfv+Wi/vF5f/uSngMiIjKEUQ9qgDuuqkwZX/g/fjsWZYiInBXGJKgh9SaYpo5udh5tHqtSREQCbcyCGuCKipPfAvOB76zDOTeG1YiIBNOYBvWzn7syZbzy3hfGqBIRkeAa06CG1C4QgEd+r2eBiIgkG/OgBtj198v6hx94cRtv7jsxhtWIiARLIII6HDK2339D//hHH17PsZbOMaxIRCQ4AhHUANGcMH++cEb/+KL7X6b2RNsYViQiEgyBCWqA79y+IGX86gdfoUkPbxKRcS5QQQ3wL59ZnDJepYc3icg4l3FQm1nYzN40s1/6WdC175nM83emXrZXsepX9PTqGmsRGZ9GckR9F7DVr0KSLZw9ka8um5/SNvdvXuDFtw+NxuZFRAIlo6A2s5nATcAP/C3npP9yzRy2/t0NKW2ff2KjHo0qIuNOpkfUDwFfBgZ9zJ2ZrTSzajOrrqury0px+blhHrzt0pS2pd9eS8WqX7HtcFNWtiEiEnTDBrWZ3Qwcdc5tGGo+59xjzrlFzrlFkydPzlqBH7tiNm997UOntN/w0Kus33Usa9sREQmqTI6orwJuNbM9wFPAdWb2E1+rGqCkIHLKreYA//H7r1Ox6ld6mJOInNNsJCFnZkuBLznnbh5qvkWLFrnq6uozLC29tduP8hc//GPaad+6rYrbr5gFx98FC0HxdAhHfKlDRCSbzGyDc25R2mlnW1ADOOcGfdJeaWEuv562mskHfgcYFE2D4hlQMgOKZ0LJzKThGVA4BUKBu5xcRMaZrAV1pvwO6j7Pb6zlr55565T2hfYnLggd4L9fXURR51Fo3A9NB6DxAHS3p84cikBxOZTMSgr0GYlA73vNnwhmvr8fERm/ztmgBjje2sXXf/4Ov3jr4LDzfuu2S5kaaePaqV3QWOuFd+3JEG+qhaaD0NudumCkYJCj8qRAj8Z8eociMh6c00Hdp6GtiwV/N/LvXnx65RJmlxUwKRYlEg5Bbw8t9QeIdR5NDfHko/KWI8CA/ZZXcupRefGMRHs0BrmFkFvkDXs/6nIREc+4CGqA5o44hbk5vLa7nk/84PUzWteVc8soi0X5xVsHKcgN88gnL2dxZSkAvd2dNB3dzzTqE2E+8Ki88QC0Hx9+I5HCpBCPQbTIe00K82jya3LQF6bOHylU8IucxcZNUA/05r4TfPTh9aOyrc9eXcmyS8vZXNvA8xsPMCnazWer8rh6dhQ6W6Crpf+1t7OZ9pZGCumAzubEtK5Wb3pz6vzxEdyJmZsc4gOCPdr3hyC5rSg17JOn50T921kicopxG9QD9fY61myo5cvPbR7rUlJMKYpytDn1ixKqZpawu66V1s4uCujkvutnU1dfz4yCHuaWOKyrleJQJ3XH6pmQ08nsWA858TboaqGjtYnujiYi3a1Ee9pS/kgQz/AZ36GIF+R9gT8w9IsHHPEXDz49UqCTsSLDUFAPoakj3v8o1R9/ZjF//exb1DWfW98u89TKJZQW5vJ2bSN7jzVy04XFnF8Cu2oPURLqpIB26urrCcWbqYi5xFF+35G+F/CdrQ1JoZ901D+wrz4dCyXCPrcw6Sc2xPhgw0njkQJ19cg5RUF9Glo6u4lFc9i0v4HaE20snD2Rfcfb+M//Us3dH7iA+381Kg8SDKyZE/O5eFqMopw4b+7Yx/JLJjC3BJZW5BGKt5ITb8V1NtHR2sTGHfuYFo0zd4Ilunj6f1pOHXeDPk7mVJGhgn+IPwJ9R//9XT/ea06ufztMZBgK6lHgnON4axdlsUTf7qs76vjUP7+RMs9186bwb9uOjkV5gfCB+VN5eesRAIqiOVx/yTTW/amO79y+gEON7Xxo/lR+vmEX331hE7fOL8HiLTQ3NfDl62ZRHOpk695DtDQ3cuWsPI6fOEFpJE4o3kpnexM9Ha1Ee9sJd7dBVyvmBb/rasV6MvwfUjiaPsCjSV0/0eKkfv2B05PaQ2Ef96ScixTUY6j2RBvTS/IJhdL30TrnMDN6eh3hkNHUESeaEyKak/hFb+qIY0C8x/HW/gbmlRdx4EQ7Ww81EQ6FWHjeBFo7e7ht9XrmTC5kd50eAztQDt0U0Ml/uryMOxZNppB2crtbeffgEWbHuonEW72TuM20tTSQ19NGqK+Lx2un0xsfeMPUYPqu6EkJ8OLUUM/rGy9JGi5OHVbgjxsK6nFsw94T3LZ6PTkh4+37ruftA43MKs3n0d/v5kfr9wBQXpLHocaOsS30LJEI/Q6KaCdm7RTSQZG1E6OdQmuniHYWT48wJRpnd+0h5pY4cnva6GptpMC1Mauwh2hPK6Gu5lNvrEqn/8i9OE2YFw/SPiD8c/J0MvcsoKCWEemI92AGO460UDmpkILcMDbIL3pXdy81Bxvp6u7l8vMmcrixg17nmDEhn437GvjOb7fzh90ZXFM+ztTc9yHe3H2IK8pzeGvnPp56tYYPzS0kRhs58WaaG4/zzu5aqiYbM/PjdLU1UhruYHKkk3h7E5F4M5HuFiyTq3hCEZwX2hYt9m7CKjp5hJ8c9GnbvNdwjv87ZhxTUMuY23a4ifMnx+judbx7rJX55cVA4o9CvKeXvfVt1LV08mcXTuFocwd/2H2cW987HUhcVtna1U0kHOL1d4+zcPYE/vGVnZQX51Eai/LFJ9/kc9fO5aZLy7nnmU3sPNoylm91VIXpIUY7RdZGEe0U0dY/HLN2ir3x5Hn62t8zwRHu69pxPcNvLFIwSMD3BX+a0M9L7u4p1h25Q1BQiwA/fbOWe55OPMTrpkvLOdTYzorFs2lqj4/zq3gc+XQSo53i/lBv7w/391fm03CinvOLeynN6eTosTqm5HbR3d4IHU1MDHeS19tKkQ3ff+8wOqyAaKyEUN/jFSIFJy+5jOR7w/neVT0FJ4cj+d544YD5vOXDuWd1F4+CWmSEnHP0OggPchI42fHWLhrb45TkR8gJG8V5kf51vP7ucaYW51E5qZDWzm62HW7ittWv+V3+mDB6E3323lF74ug+8RpLCv6+I/++tvfNyqe3q43c3nboaoN4e+KO3Ez68FMKCA0b7m1EiebHCEcLcZECXCSfUF/Ih8KJdVg4cdRv3ngo7LX1TU/XlrRMSttgy4QTf1SS2iy3YNCgVqeTSBpmRjjDg7PSwlxKC0+9BtvMWDKnrH+8MJrD5eeV8tDHFnD305v45p9fyr3Pv83Mifm8+uU/6z8P0Nge57Vdxzh/SoyGtjjLH3mNjy+ezeeunUPtiXZeqjnMdfOm0NrVzb++vo/1u+rT1rVw9gQ27msY+Zs/TY4QLRTQQsHJ+6AyOQ7cnb6572qdfDrJt04K6CSPLsrze+jqaCWfTgrMm04n+dZFQbyT/LaOxHB/e1P/ssnzR62bs+X4W0fUIuNAU0ec/Eg48YRI4DM/+uO4vqYfEv37faGdSzcXTy/iof9wKa/tPMo3X3iHL1xbyZyyAv7m+U18avFMziuNsnB2Cc++sZeqGTGefmMvdY1tTC3KpfZ4C2F6yc+BeHcPYXoJ00teDvR0d3Pte8rYevAEH3nvNH6y/l2MXnKsl3lTCpk1Icq04ghzPvo1dX2ISKr2rh66unspKUj/dXXOOTq7e9l/vI2KSYW8faCRw40dzJ0c4z1TY5gZzjn+dKSForwc8iNhQiFjy8EmvvTsW8wvL+aDF01hUizKd1/+E1fNncSxli6e21g7yu/07LD3wZsV1CISLDuONPPB767LeP6ywlzqW7t8rGhsDRXU6qMWkTFxwdQiar5xPbFoDs452rp6aO7oZlpJ3qDL7K5roSwWpSQ/9X8Bm2sbmDM5Rix6MtJ21bWwr76NxZWlFHrtrZ3dhMzIz03c8Xm0qYPFf/87H95ddumIWkTGtb4untxwiJe3HuED86fS6xytnT1EIyHM4NHf7+bOpXPJCYf45eaDnFdaSFksl67uXpo7uimfkMekWJS1249y/pQYBxs6iPf0ctX5k/q3U9fcSV4kxJ5jbazbUcenr6zo/yPlHITDIXV9iIgE2VDXUesWIRGRgFNQi4gEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYBTUIuIBJyCWkQk4BTUIiIBp6AWEQk4BbWISMApqEVEAm7YoDazWWb2ipltNbN3zOyu0ShMREQSMvkqrm7gr51zG82sCNhgZr91zm3xuTYRESGDI2rn3CHn3EZvuBnYCszwuzAREUkYUR+1mVUAlwGv+1GMiIicKuOgNrMY8Bxwt3OuKc30lWZWbWbVdXV12axRRGRcyyiozSxCIqSfcM49n24e59xjzrlFzrlFkydPzmaNIiLjWiZXfRjwz8BW59x3/C9JRESSZXJEfRXwKeA6M9vk/SzzuS4REfEMe3mec+7/AjYKtYiISBq6M1FEJOAU1CIiAaegFhEJOAW1iEjAKahFRAJOQS0iEnAKahGRgFNQi4gEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYBTUIuIBJyCWkQk4BTUIiIBp6AWEQk4BbWISMApqEVEAk5BLSIScApqEZGAU1CLiAScglpEJOAU1CIiAaegFhEJOAW1iEjAKahFRAJOQS0iEnAKahGRgFNQi4gEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwCmoRUQCLqOgNrMbzGy7me00s1V+FyUiIicNG9RmFgb+CbgRuAj4uJld5HdhIiKSkMkR9WJgp3Nut3OuC3gK+LC/ZYmISJ+cDOaZAexPGq8F/t3AmcxsJbDSG+00s5ozL88Xk4BjY13EIIJcGwS7viDXBsGuT7WdvmzWd95gEzIJakvT5k5pcO4x4DEAM6t2zi3KuLxRpNpOX5DrC3JtEOz6VNvpG636Mun6qAVmJY3PBA76U46IiAyUSVD/EbjAzCrNLBdYAfzc37JERKTPsF0fzrluM/tL4NdAGHjcOffOMIs9lo3ifKLaTl+Q6wtybRDs+lTb6RuV+sy5U7qbRUQkQHRnoohIwCmoRUSCzjmXtR/gBmA7sBNYlc11p9nWHuBtYBNQ7bWVAr8FdnivE712A77n1bUZWJi0nk978+8APp3Ufrm3/p3esjZMPY8DR4GapDbf6xlsGxnUdh9wwNt/m4BlSdPu9bazHbh+uM8XqARe92p4Gsj12qPe+E5vekWa2mYBrwBbgXeAu4Ky74aoLSj7Lg94A3jLq+8bp7vObNWdQW0/At5N2ncLxuJ3wpsvDLwJ/DIo+23QfMlicIaBXcAcINf7gC7K1vrTbG8PMGlA27f6dgqwCnjQG14GvOj9Y1gCvJ70ge72Xid6w32B8AbwPm+ZF4Ebh6nnGmAhqWHoez2DbSOD2u4DvpRm3ou8zy7q/aPa5X22g36+wDPACm/4EeDz3vCdwCPe8Arg6TTbK8f7pQSKgD95NYz5vhuitqDsOwNi3nCERAAsGek6s1l3BrX9CFie5r2M6u+EN+2vgH/lZFCP+X4bNF+yGJzvA36dNH4vcG+21p9me3s4Nai3A+VJv2TbveFHgY8PnA/4OPBoUvujXls5sC2pPWW+IWqqIDUMfa9nsG1kUNt9pA+blM+NxNU+7xvs8/V+SY4BOQP/HfQt6w3nePMN9z+TnwEfDNK+S1Nb4PYdUABsJHHX8IjWmc26M6jtR6QP6lH9XEncD/I74Drgl6fzWfi935J/stlHne5W8xlZXP9ADviNmW3wbl8HmOqcOwTgvU4Zprah2mvTtI/UaNQz2DYy8ZdmttnMHjeziadZWxnQ4JzrTlNb/zLe9EZv/rTMrAK4jMTRV6D23YDaICD7zszCZraJRNfWb0kcyY10ndmse9DanHN9++5/evvuu2YWHVhbhjWc6ef6EPBloNcbP53Pwpf9lk42gzqjW82z6Crn3EIST/X7gpldM8S8g9U20vZsCUI9q4G5wALgEPC/fKgt47rNLAY8B9ztnGsavOzR33dpagvMvnPO9TjnFpA4QlwMzD+NdfqyTwfWZmaXkDiynAdcQaI74ytZrm1YZnYzcNQ5tyG5eYj1jep+SyebQT2qt5o75w56r0eBn5L4R3rEzMoBvNejw9Q2VPvMNO0jNRr1DLaNITnnjni/SL3A90nsv9Op7RgwwcxyBrSnrMubXgIcH1iLmUVIBOETzrnnh3lfo7rv0tUWpH3XxznXAKwl0b870nVms+6harvBOXfIJXQCP+T0992ZfK5XAbea2R4STwO9jsQRdqD2W4pM+kcy+SHRd7ObRKd6Xwf6xdla/4BtFQJFScPrSZxl/QdSTyJ8yxu+idQTFW947aUkzkBP9H7eBUq9aX/05u07UbEsg7oqSO0H9r2ewbaRQW3lScP3AE95wxeTeoJkN4mTI4N+vsCzpJ4gudMb/gKpJ2GeSVOXAT8GHhrQPub7bojagrLvJgMTvOF84FXg5pGuM5t1Z1BbedK+fQh4YKx+J7zpSzl5MnHM99ugdWY5QJeRODO+C/hqNtc9YDtzvDffd+nPV732MhInCHZ4r30fqJH48oNdJC7nWZS0rs+QuIRmJ3BHUvsioMZb5h8Z/iTYkyT+Gxwn8Rf1s6NRz2DbyKC2/+NtezOJZ7ckh89Xve1sJ+lql8E+X+/zeMOr+Vkg6rXneeM7velz0tR2NYn//m0m6XK3IOy7IWoLyr6rInF52Wbv/X3tdNeZrbozqO3fvH1XA/yEk1eGjOrvRNI6lnIyqMd8vw32o1vIRUQCTncmiogEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJw/x+oC45HAf2QLwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.35454609990119934.\n",
      "Better model found at epoch 1 with accuracy value: 0.36759868264198303.\n",
      "Better model found at epoch 2 with accuracy value: 0.3754040598869324.\n",
      "Better model found at epoch 3 with accuracy value: 0.38430914282798767.\n",
      "Better model found at epoch 4 with accuracy value: 0.39266595244407654.\n",
      "Better model found at epoch 5 with accuracy value: 0.4018912613391876.\n",
      "Better model found at epoch 6 with accuracy value: 0.4131302535533905.\n",
      "Better model found at epoch 7 with accuracy value: 0.4243376851081848.\n",
      "Better model found at epoch 8 with accuracy value: 0.43364080786705017.\n",
      "Better model found at epoch 9 with accuracy value: 0.43668755888938904.\n",
      "CPU times: user 5h 48min 48s, sys: 1h 42min 19s, total: 7h 31min 7s\n",
      "Wall time: 7h 30min 9s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(10, lr, wd=wd, moms=(0.8,0.7), \n",
    "                    callbacks=[ShowGraph(learn),\n",
    "                               SaveModelCallback(learn.to_fp32(), monitor='accuracy', name='bestmodel_sp15_multifit_bwd')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "learn.to_fp32().save(mdl_path/lm_fns3_bwd[0], with_opt=False)\n",
    "learn.data.vocab.save(mdl_path/(lm_fns3_bwd[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "29ff5bf7-47d3-4bb6-8ef7-4dbee784acd0"
    }
   },
   "source": [
    "## Generate fake texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: the architecture used for our Portuguese LM is based on 4 QRNN with about 46 millions of parameters. This kind of architecture can be sufficient to fine-tune another LM to a specific corpus in order to create in-fine a text classifier (the [ULMFiT](https://nlp.fast.ai/category/classification.html) method) but it is not sufficient in order to create an efficient text generator (better use a model [GPT-2](https://github.com/openai/gpt-2) or [BERT](https://github.com/google-research/bert)). More, the SentencePiece tokenizer used in this notebook implements subword units (e.g., byte-pair-encoding (BPE) [Sennrich et al.]) and unigram language model [Kudo.]) that can generate caracters from its vocabulary instead of words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "nbpresent": {
     "id": "903b31b8-77bb-48a7-a6de-fd2584005619"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.11 s, sys: 1.81 s, total: 4.92 s\n",
      "Wall time: 13.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15_multifit', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "nbpresent": {
     "id": "f2e132d0-a490-49cb-895a-ee9a66c76c2d"
    }
   },
   "outputs": [],
   "source": [
    "# LM without pretraining\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, pretrained=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "nbpresent": {
     "id": "7596a7ac-558a-4bd7-80af-247bf0a82732"
    }
   },
   "outputs": [],
   "source": [
    "# LM pretrained in English\n",
    "learn_en = language_model_learner(data, AWD_LSTM, pretrained=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "nbpresent": {
     "id": "cdaac260-7cfc-4255-b03f-e1a88eccdc71"
    }
   },
   "outputs": [],
   "source": [
    "# LM pretrained in french\n",
    "learn_pt = language_model_learner(data, AWD_LSTM, config=config, pretrained_fnames=lm_fns3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "nbpresent": {
     "id": "12d9ce7c-fdb1-4108-ab95-764a015f8e0e"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal venceu o torneio ▁comprar ▁relevante ▁cordas mila ▁lutadores ▁assumido fos ল velmente ▁conjunto ▁american ▁lentamente ▁empregada ▁extra เ anfitriã ⵙ ▁gonzaga ⊂ ̌ coli ▁feijão gi ▁útil ▁notável evolution my 20 ▁gen ▁subespécie ổ ▁cachoeira ▁natal nik ▁jornal ▁cartões ▁desportivo ▁grega ỏ sa ▁azulejos ▁habitat ▁energia fc ▁fazê 68 ▁reencontra ▁humorístico ▁mitchell ▁kris organiz ▁designar ▁castro ▁cerâmica ▁bob burg dos ▁usualmente ▁pessoas ▁principais ▁saudita ▁rom ▁escolhido ▁abandonou ▁patrimônio rid ib ▁ventos ▁banda ▁fim ▁dinamarquesa ን ▁divididos ▁apropriado carna ▁graves ▁vir ▁fm ខ far ▁cbf 提 wel ▁aclamação ├ 童 ▁vinda ▁alusão mun ▁bacia 工 ▁arrenda ▁biológica ▁marginal ▁cheio hí ▁chicago ▁daí 화 ▁parceira\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal venceu o torneio\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "nbpresent": {
     "id": "9be3d15c-526a-4ab7-bd46-16e8a8c58142"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal venceu o torneio into the \" big - screen \" . a \" ' cop - man ' game , \" \" a first - time \" , is a one - game , game - to - day play . \" a \" as a game for the \" bad man \" , \" you - do - it - you - do - it - me - to - do \" game , \" the first game \" , is \" the first game in the world \" . a game in all four game , \" the game \" ,\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal venceu o torneio\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn_en.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "nbpresent": {
     "id": "35c213f4-3c90-4774-9cdd-56da02ebe3a8"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal venceu o torneio ▁xxmaj ▁ ▁oficial ▁de ▁ , ▁em ▁seu ▁segundo ▁dia ▁no ▁dia ▁da ▁xxmaj ▁copa ▁xxmaj ▁ master s , ▁em ▁xxmaj ▁ las ▁xxmaj ▁vegas , ▁em ▁15 ▁de ▁xxmaj ▁abril ▁de ▁2016. ▁xxmaj ▁durante ▁a ▁temporada , ▁foi ▁o ▁primeiro ▁título ▁da ▁liga , ▁vencendo ▁o ▁xxmaj ▁ s pur s ▁por ▁9 zo s ▁por ▁3 ▁a ▁0 . ▁o ▁campeão ▁foi ▁o ▁ , ▁da ▁xxmaj ▁copa ▁do ▁xxmaj ▁mundo ▁de ▁xxmaj ▁ ty r rell , ▁onde ▁o ▁título ▁passou ▁a ▁ser ▁xxmaj ▁ s qua d . ▁a ▁xxup ▁ m l s ▁é\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal venceu o torneio\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn_pt.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}