seminars/2_Language_Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XgrrPZLc44t4"
   },
   "source": [
    "# https://clck.ru/JC6Ea"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7Jm3LqcY3Io4"
   },
   "source": [
    "## Препроцессинг и определение языка / Preprocessing and language detection\n",
    "\n",
    "Предыдущая часть была посвящена предобработке текстовых данных, теперь применим эти методы на практике и заодно узнаем, как можно решать задачу определения языка.\n",
    "\n",
    "**Идеи?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OFMHW4Ci3Io5"
   },
   "source": [
    "Нам понадобятся некоторые пакеты для Питона, установите себе недостающие:\n",
    "* wikipedia\n",
    "* nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6236,
     "status": "ok",
     "timestamp": 1568988329195,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "v9kJxTkI3Io6",
    "outputId": "fe87504b-d59e-4876-e1f6-d9b3b0cce697"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting wikipedia\n",
      "  Downloading https://files.pythonhosted.org/packages/67/35/25e68fbc99e672127cc6fbb14b8ec1ba3dfef035bf1e4c90f78f24a80b7d/wikipedia-1.4.0.tar.gz\n",
      "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.6/dist-packages (from wikipedia) (4.6.3)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from wikipedia) (2.21.0)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2.8)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (1.24.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2019.6.16)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.0.4)\n",
      "Building wheels for collected packages: wikipedia\n",
      "  Building wheel for wikipedia (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for wikipedia: filename=wikipedia-1.4.0-cp36-none-any.whl size=11686 sha256=4e6843412b730a2622029e5d38490f50ba0c970dff8545106c4d6bce30d8c6c9\n",
      "  Stored in directory: /root/.cache/pip/wheels/87/2a/18/4e471fd96d12114d16fe4a446d00c3b38fb9efcb744bd31f4a\n",
      "Successfully built wikipedia\n",
      "Installing collected packages: wikipedia\n",
      "Successfully installed wikipedia-1.4.0\n"
     ]
    }
   ],
   "source": [
    "!pip install wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8704,
     "status": "ok",
     "timestamp": 1568988332164,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "urgPFXaE3Io8",
    "outputId": "62515258-9aeb-491e-b01e-dfa56f2473fd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (3.2.5)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk) (1.12.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IzpLP1ve3Io-"
   },
   "outputs": [],
   "source": [
    "from collections import Counter  # нужно объяснять, что это?\n",
    "\n",
    "import nltk\n",
    "import wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VZM66tLa3IpA"
   },
   "source": [
    "Возможно, перед первым запуском `nltk` нужно скачать немного данных:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 32889,
     "status": "ok",
     "timestamp": 1568988577522,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "tCpai7Yc3IpA",
    "outputId": "c26be02a-4066-44cd-be01-43c7887ba703"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NLTK Downloader\n",
      "---------------------------------------------------------------------------\n",
      "    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit\n",
      "---------------------------------------------------------------------------\n",
      "Downloader> d\n",
      "\n",
      "Download which package (l=list; x=cancel)?\n",
      "  Identifier> punkt\n",
      "    Downloading package punkt to /root/nltk_data...\n",
      "      Unzipping tokenizers/punkt.zip.\n",
      "\n",
      "---------------------------------------------------------------------------\n",
      "    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit\n",
      "---------------------------------------------------------------------------\n",
      "Downloader> q\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download(\"punkt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ABmBn1Sz3IpC"
   },
   "source": [
    "Теперь посмотрим, как можно скачивать тексты из Википедии с помощью пакета `wikipedia`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 258
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1611,
     "status": "ok",
     "timestamp": 1568988769838,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "ZsBAih1d3IpC",
    "outputId": "99a40886-4b8b-436f-90b1-a73ee32f8cd1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Генделевский фестиваль в Гёттингене\n",
      "Генделевский фестиваль в Гёттингене\n",
      "Генделевский фестиваль в Гёттингене (нем. Internationale Händel-Festspiele Göttingen) — ежегодный фестиваль старинной музыки в Гёттингене, посвящённый в первую очередь музыке Георга Фридриха Генделя. Программа фестиваля длится не менее двух дней, в один из которых ставится одна из опер Генделя, в другой проводится концерт с исполнением нескольких его ораторий наряду с так или иначе перекликающимися с ними сочинениями современников композитора.\n",
      "Фестиваль был основан в 1920 году искусствоведом Оскаром Хагеном, по инициативе которого была впервые после почти двухсотлетнего перерыва поставлена опера Генделя «Роделинда». Под руководством Хагена в начале 1920-х гг. эстетика постановок была отмечена сильным влиянием экспрессионизма (первым сценографом фестиваля выступал Пауль Тирш), а музыка подвергалась значительным коррективам. Ранняя критика окрестила фестиваль «Генделевским Байройтом». После кризиса на рубеже десятилетий (в 1928—1933 гг. фестиваль не проводился) акцент заметно сместился в сторону исторической корректности, особенно значительным был фестиваль 1937 года, на котором приглашённым дирижёром выступил руководитель Музыкальной палаты Третьего Рейха Петер Раабе.\n",
      "В разные годы среди художественных руководителей фестиваля были Фриц Леман (1934—1953), Джон Элиот Гардинер (1981—1990), Николас Макгеган (1991—2011), Лоуренс Каммингс (с 2012 г.).\n",
      "Международный авторитет фестиваля значительно упрочило появление начиная с 1990-х гг. многочисленных записей опер и ораторий, сделанных непосредственно на фестивале или на основе подготовленных для фестиваля исполнений.\n",
      "\n",
      "\n",
      "== Примечания ==\n",
      "\n",
      "\n",
      "== Ссылки ==\n",
      "Официальный сайт\n"
     ]
    }
   ],
   "source": [
    "wikipedia.set_lang('ru')\n",
    "page_name = wikipedia.random(1)\n",
    "page = wikipedia.page(page_name)\n",
    "print(page_name)\n",
    "print(page.title)\n",
    "print(page.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MQRQF5n93IpE"
   },
   "source": [
    "Поскольку мы будем выбирать из нескольких языков, лучше сразу завести функцию, которая будет \n",
    "скачивать нужное количество страниц для заданного языка."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uZ0EfvHG3IpF"
   },
   "outputs": [],
   "source": [
    "TEST_LANGS = ('kk', 'ru', 'uk', 'be', 'fr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yA5fu5vo3IpG"
   },
   "outputs": [],
   "source": [
    "def get_texts_for_lang(lang, n=10):\n",
    "    wiki_content = []\n",
    "    wikipedia.set_lang(lang)\n",
    "    pages = wikipedia.random(n)\n",
    "    for page_name in pages:\n",
    "        try:\n",
    "            page = wikipedia.page(page_name)\n",
    "            wiki_content.append(\"%s\\n%s\" % (page.title, page.content.replace('=', '')))\n",
    "        except wikipedia.exceptions.WikipediaException:\n",
    "            print(\"Skip %s\" % page_name)\n",
    "    return wiki_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 180193,
     "status": "ok",
     "timestamp": 1568989351776,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "6a1XLXXb3IpI",
    "outputId": "ae836aaf-325f-41ea-b006-6b6152590863"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kk 50\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/wikipedia/wikipedia.py:389: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"lxml\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "The code that caused this warning is on line 389 of the file /usr/local/lib/python3.6/dist-packages/wikipedia/wikipedia.py. To get rid of this warning, pass the additional argument 'features=\"lxml\"' to the BeautifulSoup constructor.\n",
      "\n",
      "  lis = BeautifulSoup(html).find_all('li')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skip Ясеновское сельское поселение\n",
      "Skip Урбар\n",
      "Skip Холмец\n",
      "ru 47\n",
      "Skip Норт-Адамс\n",
      "Skip 1574 (значення)\n",
      "Skip Архангельська сільська рада\n",
      "Skip Мара (значення)\n",
      "uk 46\n",
      "Skip Макараўцы\n",
      "Skip Паўлаўцы\n",
      "Skip Карака (значэнні)\n",
      "Skip Ян Сапега\n",
      "be 46\n",
      "fr 50\n"
     ]
    }
   ],
   "source": [
    "wiki_texts = {}\n",
    "for lang in TEST_LANGS:\n",
    "    wiki_texts[lang] = get_texts_for_lang(lang, 50)\n",
    "    print(lang, len(wiki_texts[lang]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 530
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 568,
     "status": "ok",
     "timestamp": 1568989396257,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "ZQmCT_J23IpL",
    "outputId": "cecab675-0382-4d17-c193-435a1acc0be5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Барақ (ит)\n",
      "Барақ - асыл текті ит тұқымының бейнелі атауы. Ілкі дәуірлерге тән археологиялық қазбаларда итті адам мүрдесінің жанына жерлегені мәлім. Бұл итті жеті қазынаға жатқызатын наным-сенімнің көне бастауы болса керек. Ежелгі түсінікте барақ пен бөрінің жігі ажыратыла бермеген, екеуі де тотемдік ұғыммен байланысты болғандықтан, көне рәміздерде, нышандық белгілерде ит-қасқыр бейнелері түркі-моңғол халықтарында кеңінен көрініс тапқан. Мәселен, «Моңғолдың құпия шежіресінде» Шыңғысханның арғы анасы Алунғуаның жатырына түнде шаңырақтан сары нұр түсіп, ол таң бозаландағанда есіктен сары ит түрінде сағымданып шығып кеткендігі,содан ананың жүкті болып қалғаны айтылады. Сонымен бірге, қырғыз халқының арғы тегі қыз бен қызыл иттің жұптасуынан таралған деген аңыз бар. Түркі халықтары балаларына барақ деген атты көп қойған. Батырларын Бараққа теңеген. Дүрегей деген ит тұқымының еркегі барақ деп аталатыны жөнінде де біршама деректер сақталған.\n",
      "\n",
      "\n",
      " Дереккөздер \n",
      "Thomas Ulmer\n",
      "Thomas Ulmer est un député européen allemand né le 25 juillet 1956 à Karlsruhe. Il est membre de l'Union chrétienne-démocrate d'Allemagne.\n",
      "\n",
      "\n",
      " Biographie \n",
      "Il a été élu député européen lors des élections européennes de 2004.\n",
      "Au cours de la 7e législature, il siège au sein du Parti populaire européen. Il est vice-président de la Délégation pour les relations avec la Suisse et la Norvège, à la commission parlementaire mixte UE-Islande et à la commission parlementaire mixte de l'Espace économique européen (EEE) depuis 2009.\n",
      "\n",
      "\n",
      " Notes et références \n",
      "\n",
      "\n",
      " Annexes \n",
      "\n",
      "\n",
      " Articles connexes \n",
      "\n",
      "Liste des députés européens d'Allemagne de la 7e législature\n",
      "Liste des députés européens d'Allemagne de la 6e législature\n",
      "\n",
      "\n",
      " Liens externes \n",
      "\n",
      "Fiche de présentation sur le site du Parlement européen. Portail de la politique   Portail de l’Allemagne   Portail de l’Union européenne\n"
     ]
    }
   ],
   "source": [
    "print(wiki_texts['kk'][0])\n",
    "print(wiki_texts['fr'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UBOLiySe3IpN"
   },
   "source": [
    "**Идея 1**\n",
    "\n",
    "Можно найти самые частотные слова для каждого языка, а потом считать их количество в каждом новом тексте."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5SqFWksT3IpN"
   },
   "outputs": [],
   "source": [
    "def collect_freqlist(wiki_pages, max_len=100):\n",
    "    freqlist = Counter()\n",
    "    # не забудем про токенизацию - nltk.word_tokenize\n",
    "    for text in wiki_pages:\n",
    "        for word in nltk.word_tokenize(text.lower()):\n",
    "            if word.isalpha():\n",
    "                freqlist[word] += 1\n",
    "    return dict(freqlist.most_common(max_len))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 986,
     "status": "ok",
     "timestamp": 1568990123911,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "Yrb4JNpk3IpQ",
    "outputId": "17cb66b1-8060-40a0-8048-5dfa1892bf75"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'isbn': 21,\n",
       " 'me': 18,\n",
       " 'the': 39,\n",
       " 'а': 74,\n",
       " 'альбом': 31,\n",
       " 'англ': 18,\n",
       " 'был': 97,\n",
       " 'была': 35,\n",
       " 'были': 43,\n",
       " 'было': 40,\n",
       " 'в': 1055,\n",
       " 'вместе': 16,\n",
       " 'во': 34,\n",
       " 'время': 34,\n",
       " 'где': 26,\n",
       " 'год': 19,\n",
       " 'года': 192,\n",
       " 'годах': 18,\n",
       " 'году': 106,\n",
       " 'два': 17,\n",
       " 'движения': 29,\n",
       " 'деревня': 29,\n",
       " 'деятельность': 18,\n",
       " 'для': 60,\n",
       " 'до': 57,\n",
       " 'его': 97,\n",
       " 'её': 19,\n",
       " 'же': 22,\n",
       " 'жизни': 16,\n",
       " 'за': 77,\n",
       " 'и': 643,\n",
       " 'из': 131,\n",
       " 'или': 27,\n",
       " 'им': 20,\n",
       " 'имени': 19,\n",
       " 'искусства': 16,\n",
       " 'истории': 23,\n",
       " 'история': 21,\n",
       " 'их': 19,\n",
       " 'к': 106,\n",
       " 'как': 82,\n",
       " 'км': 29,\n",
       " 'константин': 17,\n",
       " 'которого': 17,\n",
       " 'которые': 16,\n",
       " 'который': 21,\n",
       " 'которых': 20,\n",
       " 'культуры': 19,\n",
       " 'магнитудой': 20,\n",
       " 'марта': 19,\n",
       " 'михаила': 19,\n",
       " 'москве': 16,\n",
       " 'на': 350,\n",
       " 'не': 106,\n",
       " 'несколько': 16,\n",
       " 'николас': 16,\n",
       " 'но': 35,\n",
       " 'нод': 62,\n",
       " 'о': 55,\n",
       " 'области': 26,\n",
       " 'однако': 27,\n",
       " 'около': 18,\n",
       " 'он': 71,\n",
       " 'от': 88,\n",
       " 'по': 197,\n",
       " 'под': 31,\n",
       " 'получил': 16,\n",
       " 'после': 46,\n",
       " 'при': 37,\n",
       " 'примечания': 34,\n",
       " 'пушкина': 26,\n",
       " 'режиссёр': 28,\n",
       " 'реки': 27,\n",
       " 'роль': 16,\n",
       " 'россии': 28,\n",
       " 'с': 276,\n",
       " 'сайте': 18,\n",
       " 'своей': 23,\n",
       " 'со': 23,\n",
       " 'состав': 16,\n",
       " 'ссылки': 27,\n",
       " 'так': 19,\n",
       " 'также': 66,\n",
       " 'театр': 18,\n",
       " 'то': 16,\n",
       " 'того': 18,\n",
       " 'том': 17,\n",
       " 'у': 26,\n",
       " 'февраля': 16,\n",
       " 'части': 16,\n",
       " 'часть': 20,\n",
       " 'человек': 28,\n",
       " 'через': 20,\n",
       " 'что': 89,\n",
       " 'шакур': 46,\n",
       " 'шакура': 31,\n",
       " 'это': 25,\n",
       " 'этом': 19,\n",
       " 'я': 19,\n",
       " 'является': 32}"
      ]
     },
     "execution_count": 22,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# проверка\n",
    "collect_freqlist(wiki_texts['ru'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PVCHWvJ53IpR"
   },
   "outputs": [],
   "source": [
    "freq_lists = {}\n",
    "for lang in TEST_LANGS:\n",
    "    freq_lists[lang] = collect_freqlist(wiki_texts[lang])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "twrmWoH-3IpT"
   },
   "source": [
    "Теперь всё готово, чтобы сделать первую определялку языка!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s303QEYg3IpT"
   },
   "outputs": [],
   "source": [
    "def simple_lang_detect(freq_lists, text):\n",
    "    counts = Counter()\n",
    "    for lang, freq_list in freq_lists.items():\n",
    "        freq_list = Counter(freq_list)\n",
    "        for word in nltk.word_tokenize(text):\n",
    "            counts[lang] += int(freq_list[word] > 0)\n",
    "    return counts.most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 428
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 7533,
     "status": "ok",
     "timestamp": 1568990404963,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "R3CtzkM73IpV",
    "outputId": "37ae567b-10c4-4599-f8dd-01b8dd2c64fc"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/wikipedia/wikipedia.py:389: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"lxml\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "The code that caused this warning is on line 389 of the file /usr/local/lib/python3.6/dist-packages/wikipedia/wikipedia.py. To get rid of this warning, pass the additional argument 'features=\"lxml\"' to the BeautifulSoup constructor.\n",
      "\n",
      "  lis = BeautifulSoup(html).find_all('li')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skip Głogówko\n",
      "Rovescala\n",
      "Rovescala est une commune italienne de la province de Pavie dans la région Lombardie en Italie.\n",
      "\n",
      "\n",
      " Administration \n",
      "\n",
      "\n",
      " Hameaux \n",
      "\n",
      "\n",
      " Communes limitrophes \n",
      "Castel San Giovanni, Montù Beccaria, San Damiano al Colle, Santa Maria della Versa, Ziano Piacentino\n",
      "\n",
      "\n",
      " Notes et références \n",
      "\n",
      " Portail de la Lombardie\n"
     ]
    }
   ],
   "source": [
    "test_texts = get_texts_for_lang('fr')[0]\n",
    "print(test_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 362,
     "status": "ok",
     "timestamp": 1568990423832,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "u6Z7VdXI3IpW",
    "outputId": "a3dc58a6-9785-44fe-c82d-8c29bb2e9bd4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('fr', 12), ('kk', 3), ('ru', 0), ('uk', 0), ('be', 0)]"
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_lang_detect(freq_lists, test_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fnfJHW843IpY"
   },
   "source": [
    "Попробуем оценить качество нашей определялки. Для этого установим `sklearn`, если его ещё нет, и возьмем оттуда функцию для подсчёта accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3685,
     "status": "ok",
     "timestamp": 1568990559267,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "y9kUURxH3IpY",
    "outputId": "f150d9a6-b47b-4e89-d6f6-832579861799"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (0.0)\n",
      "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sklearn) (0.21.3)\n",
      "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn) (1.3.1)\n",
      "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn) (0.13.2)\n",
      "Requirement already satisfied: numpy>=1.11.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn) (1.16.5)\n"
     ]
    }
   ],
   "source": [
    "!pip install sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Q25dDPmw3IpZ"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3ullc9-O3Ipb"
   },
   "outputs": [],
   "source": [
    "def test_simple_lang_detect(freq_lists, test_size):\n",
    "    results = []  # сюда будем писать результаты\n",
    "    gold = []     # сюда будем писать исходный язык\n",
    "    for lang in TEST_LANGS:\n",
    "        for text in get_texts_for_lang(lang, test_size):\n",
    "            predicted_lang = simple_lang_detect(freq_lists, text)[0][0]\n",
    "            results.append(predicted_lang)\n",
    "            gold.append(lang)\n",
    "    print(\"RESULTS:\")\n",
    "    print(\"%d languages\" % len(TEST_LANGS))\n",
    "    print(\"Test size: %d texts per language\" % test_size)\n",
    "    print(\"Accuracy: %.4f\" % accuracy_score(results, gold))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 36878,
     "status": "ok",
     "timestamp": 1568990818821,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "MDavJybX3Ipd",
    "outputId": "97dad5d4-7ab3-4ace-f63e-22d2bbe405dc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skip Виноградовское сельское поселение (Приморский край)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/wikipedia/wikipedia.py:389: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"lxml\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "The code that caused this warning is on line 389 of the file /usr/local/lib/python3.6/dist-packages/wikipedia/wikipedia.py. To get rid of this warning, pass the additional argument 'features=\"lxml\"' to the BeautifulSoup constructor.\n",
      "\n",
      "  lis = BeautifulSoup(html).find_all('li')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skip Сайра\n",
      "Skip Вячорка\n",
      "RESULTS:\n",
      "5 languages\n",
      "Test size: 10 texts per language\n",
      "Accuracy: 0.9574\n"
     ]
    }
   ],
   "source": [
    "# проверка\n",
    "test_simple_lang_detect(freq_lists, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TiXZGRXM3Ipf"
   },
   "source": [
    "## Машинное обучение\n",
    "\n",
    "Будем говорить о **supervised** методах (обучение с учителем):\n",
    "\n",
    "* У нас есть пары `(признаки, класс)`\n",
    "* Классификатор обучается на них — подбирает подходящую функцию отображения признаков в множество классов\n",
    "* После этого можно применять полученную модель для предсказаний на новых данных\n",
    "\n",
    "Признаки могут быть разные:\n",
    "* средняя длина слова\n",
    "* минимальная длина слова\n",
    "* максимальная длина слова\n",
    "* …"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MWOfXYQb3Ipg"
   },
   "source": [
    "Для определения языка часто используют не слова, а последовательности символов (символьные энграммы / character ngrams). Например, 3-граммы:\n",
    "```\n",
    "сим, имв, мво, вол, оло, лов\n",
    "```\n",
    "\n",
    "Из обучающих данных соберем словарь символьных энграмм $V$. \n",
    "Тогда каждый текст сможем представить в виде вектора длины $|V|$, где каждый признак показывает, присутствует ли соответствующая энграмма в тексте. Если эти значения будут просто $0$/$1$, то не учитывается \"важность\" последовательности для языка.\n",
    "\n",
    "Будем использовать $tf \\cdot idf$:\n",
    "\n",
    "$tf \\cdot idf (n, d) = \\frac{count(n_d)}{\\sum_{w \\in d}count(w)} \\cdot log\\frac{|D|}{|\\{d \\in D | n \\in d\\}|}$\n",
    "\n",
    "где $n$ - энграмма, $d$ - документ, а $D$ - весь корпус (на данном языке)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1881,
     "status": "ok",
     "timestamp": 1568991119503,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "QYCDVKsX3Iph",
    "outputId": "624ee149-b748-41ab-f7d0-1c908c7d0a3b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      ".\n",
      "\n",
      ". \n",
      "\n",
      ". д\n",
      "\n",
      ". дл\n",
      "\n",
      "1\n",
      "\n",
      "11\n",
      "\n",
      "11 \n",
      "\n",
      "11 д\n",
      "\n",
      "12\n",
      "\n",
      "12 \n",
      "\n",
      "12 м\n",
      "\n",
      "19\n",
      "\n",
      "196\n",
      "\n",
      "1963\n",
      "\n",
      "1964\n",
      "\n",
      "1965\n",
      "\n",
      "1966\n",
      "\n",
      "1968\n",
      "\n",
      "1969\n",
      "\n",
      "197\n",
      "\n",
      "1970\n",
      "\n",
      "1971\n",
      "\n",
      "1972\n",
      "\n",
      "1973\n",
      "\n",
      "1974\n",
      "\n",
      "1975\n",
      "\n",
      "1976\n",
      "\n",
      "1978\n",
      "\n",
      "198\n",
      "\n",
      "1980\n",
      "\n",
      "1984\n",
      "\n",
      "1985\n",
      "\n",
      "199\n",
      "\n",
      "1995\n",
      "\n",
      "1996\n",
      "\n",
      "1997\n",
      "\n",
      "1998\n",
      "\n",
      "1999\n",
      "\n",
      "2\n",
      "\n",
      "20\n",
      "\n",
      "20 \n",
      "\n",
      "20 и\n",
      "\n",
      "200\n",
      "\n",
      "2001\n",
      "\n",
      "2002\n",
      "\n",
      "2003\n",
      "\n",
      "2004\n",
      "\n",
      "2005\n",
      "\n",
      "2006\n",
      "\n",
      "2007\n",
      "\n",
      "2008\n",
      "\n",
      "2009\n",
      "\n",
      "201\n",
      "\n",
      "2010\n",
      "\n",
      "2011\n",
      "\n",
      "2012\n",
      "\n",
      "2013\n",
      "\n",
      "2014\n",
      "\n",
      "2015\n",
      "\n",
      "2016\n",
      "\n",
      "2017\n",
      "\n",
      "2018\n",
      "\n",
      "2019\n",
      "\n",
      "25\n",
      "\n",
      "25-\n",
      "\n",
      "25-л\n",
      "\n",
      "26\n",
      "\n",
      "26 \n",
      "\n",
      "26 и\n",
      "\n",
      "4\n",
      "\n",
      "4 \n",
      "\n",
      "4 м\n",
      "\n",
      "4 ма\n",
      "\n",
      "46\n",
      "\n",
      "46-\n",
      "\n",
      "46-й\n",
      "\n",
      "8\n",
      "\n",
      "8 \n",
      "\n",
      "8 ф\n",
      "\n",
      "8 фе\n",
      "\n",
      "9\n",
      "\n",
      "9 \n",
      "\n",
      "9 а\n",
      "\n",
      "9 ап\n",
      "\n",
      "9 н\n",
      "\n",
      "9 но\n",
      "\n",
      "a\n",
      "\n",
      "al\n",
      "\n",
      "all\n",
      "\n",
      "all \n",
      "\n",
      "b\n",
      "\n",
      "be\n",
      "\n",
      "bet\n",
      "\n",
      "bett\n",
      "\n",
      "c\n",
      "\n",
      "c \n",
      "\n",
      "c 1\n",
      "\n",
      "c 1 \n",
      "\n",
      "cg\n"
     ]
    }
   ],
   "source": [
    "from sklearn import feature_extraction\n",
    "\n",
    "vectorizer = feature_extraction.text.TfidfVectorizer(ngram_range=(1, 5), analyzer='char')\n",
    "vectorizer.fit(wiki_texts['ru'])\n",
    "for item in vectorizer.get_feature_names()[:100]:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 884
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1197,
     "status": "ok",
     "timestamp": 1568991145252,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "7WHMuiAF3Ipj",
    "outputId": "da74f788-4654-43cf-ea42-f8dcf410abfc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 128664)\t0.0007484019916797908\n",
      "  (0, 128663)\t0.0007484019916797908\n",
      "  (0, 128662)\t0.0007484019916797908\n",
      "  (0, 128661)\t0.0007484019916797908\n",
      "  (0, 128650)\t0.0007484019916797908\n",
      "  (0, 128649)\t0.0007484019916797908\n",
      "  (0, 128648)\t0.0007484019916797908\n",
      "  (0, 128636)\t0.0005516109357697634\n",
      "  (0, 128631)\t0.0010479967928373186\n",
      "  (0, 128630)\t0.0007484019916797908\n",
      "  (0, 128629)\t0.0007484019916797908\n",
      "  (0, 128628)\t0.0007484019916797908\n",
      "  (0, 128627)\t0.0007484019916797908\n",
      "  (0, 128622)\t0.0007484019916797908\n",
      "  (0, 128621)\t0.0007484019916797908\n",
      "  (0, 128620)\t0.0007484019916797908\n",
      "  (0, 128616)\t0.0006757722681801439\n",
      "  (0, 128600)\t0.0007484019916797908\n",
      "  (0, 128599)\t0.0007484019916797908\n",
      "  (0, 128596)\t0.0006757722681801439\n",
      "  (0, 128592)\t0.0006242406592694103\n",
      "  (0, 128588)\t0.0007484019916797908\n",
      "  (0, 128587)\t0.0007484019916797908\n",
      "  (0, 128586)\t0.0007484019916797908\n",
      "  (0, 128585)\t0.0006757722681801439\n",
      "  :\t:\n",
      "  (0, 154)\t0.0007484019916797908\n",
      "  (0, 153)\t0.0007484019916797908\n",
      "  (0, 152)\t0.0006757722681801439\n",
      "  (0, 151)\t0.0006757722681801439\n",
      "  (0, 147)\t0.0007484019916797908\n",
      "  (0, 146)\t0.0007484019916797908\n",
      "  (0, 145)\t0.0007484019916797908\n",
      "  (0, 144)\t0.0006757722681801439\n",
      "  (0, 143)\t0.0007484019916797908\n",
      "  (0, 142)\t0.0007484019916797908\n",
      "  (0, 139)\t0.0006757722681801439\n",
      "  (0, 133)\t0.0006242406592694103\n",
      "  (0, 128)\t0.0007484019916797908\n",
      "  (0, 127)\t0.0007484019916797908\n",
      "  (0, 126)\t0.0007484019916797908\n",
      "  (0, 125)\t0.0007484019916797908\n",
      "  (0, 94)\t0.0007484019916797908\n",
      "  (0, 93)\t0.0007484019916797908\n",
      "  (0, 92)\t0.0007484019916797908\n",
      "  (0, 91)\t0.0007484019916797908\n",
      "  (0, 90)\t0.0007484019916797908\n",
      "  (0, 89)\t0.0007484019916797908\n",
      "  (0, 88)\t0.0007484019916797908\n",
      "  (0, 87)\t0.0007484019916797908\n",
      "  (0, 0)\t0.0026869040780793303\n"
     ]
    }
   ],
   "source": [
    "print(vectorizer.transform(wiki_texts['ru'])[0])  # первый документ в векторном представлении"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6277,
     "status": "ok",
     "timestamp": 1568991189618,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "CTmIMc9A3Ipl",
    "outputId": "d6109adf-ec1c-4222-f9d1-593df16727d4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: matplotlib==3.0.3 in /usr/local/lib/python3.6/dist-packages (3.0.3)\n",
      "Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.0.3) (1.16.5)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.0.3) (2.5.3)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.0.3) (0.10.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.0.3) (2.4.2)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.0.3) (1.1.0)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.1->matplotlib==3.0.3) (1.12.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib==3.0.3) (41.2.0)\n",
      "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (0.9.0)\n",
      "Requirement already satisfied: matplotlib>=1.4.3 in /usr/local/lib/python3.6/dist-packages (from seaborn) (3.0.3)\n",
      "Requirement already satisfied: numpy>=1.9.3 in /usr/local/lib/python3.6/dist-packages (from seaborn) (1.16.5)\n",
      "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn) (0.24.2)\n",
      "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn) (1.3.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn) (0.10.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn) (1.1.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn) (2.4.2)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn) (2.5.3)\n",
      "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn) (2018.9)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib>=1.4.3->seaborn) (1.12.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib>=1.4.3->seaborn) (41.2.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install matplotlib==3.0.3\n",
    "!pip install seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 569,
     "status": "ok",
     "timestamp": 1568991192947,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "i0ErpduK3Ipn",
    "outputId": "41904f7f-c2f2-42ff-bbd9-3eedcb6bd377"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "from sklearn import pipeline\n",
    "from sklearn import naive_bayes\n",
    "import numpy as np\n",
    "\n",
    "%pylab inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ManAEc9n3Ipo"
   },
   "outputs": [],
   "source": [
    "clf = pipeline.Pipeline([\n",
    "    ('vctr', feature_extraction.text.TfidfVectorizer(ngram_range=(1, 2), analyzer='char')),\n",
    "    ('clf', naive_bayes.MultinomialNB())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "P6h0iZMI3Ipq"
   },
   "outputs": [],
   "source": [
    "all_texts = []\n",
    "lang_indices = []\n",
    "for lang in wiki_texts:\n",
    "    all_texts.extend(wiki_texts[lang])\n",
    "    lang_indices.extend([lang]*len(wiki_texts[lang]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1039,
     "status": "ok",
     "timestamp": 1568991240937,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "cCvMe95v3Ips",
    "outputId": "b30dd417-ba90-4bb3-a475-ceb52d23f460"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('vctr',\n",
       "                 TfidfVectorizer(analyzer='char', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.float64'>,\n",
       "                                 encoding='utf-8', input='content',\n",
       "                                 lowercase=True, max_df=1.0, max_features=None,\n",
       "                                 min_df=1, ngram_range=(1, 2), norm='l2',\n",
       "                                 preprocessor=None, smooth_idf=True,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 sublinear_tf=False,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, use_idf=True,\n",
       "                                 vocabulary=None)),\n",
       "                ('clf',\n",
       "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 40,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Обучаем классификатор\n",
    "clf.fit(np.array(all_texts), np.array(lang_indices))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 391
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 855,
     "status": "ok",
     "timestamp": 1568991261666,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "Vk-znp6f3Ipt",
    "outputId": "a495ed4c-bdcc-4eb1-86f5-02c7192dce4d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk',\n",
       "       'kk', 'fr', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk',\n",
       "       'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk',\n",
       "       'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'kk',\n",
       "       'kk', 'kk', 'kk', 'kk', 'kk', 'kk', 'ru', 'ru', 'ru', 'ru', 'ru',\n",
       "       'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru',\n",
       "       'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru',\n",
       "       'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru',\n",
       "       'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'ru', 'uk', 'uk',\n",
       "       'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk',\n",
       "       'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk',\n",
       "       'uk', 'uk', 'uk', 'fr', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk',\n",
       "       'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk', 'uk',\n",
       "       'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be',\n",
       "       'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be',\n",
       "       'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'ru',\n",
       "       'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be',\n",
       "       'be', 'be', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr',\n",
       "       'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr',\n",
       "       'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr',\n",
       "       'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr',\n",
       "       'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr', 'fr'], dtype='<U2')"
      ]
     },
     "execution_count": 41,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Предсказываем результаты для тех же текстов\n",
    "clf.predict(all_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 7588,
     "status": "ok",
     "timestamp": 1568991284256,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "bM-uxf993Ipv",
    "outputId": "525d221a-d03f-4891-842d-5375b4c4a729"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be', 'be'],\n",
       "      dtype='<U2')"
      ]
     },
     "execution_count": 42,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Скачиваем новые тексты из вики и предсказываем еще раз\n",
    "clf.predict(get_texts_for_lang('be'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DeoUXwXe3Ipw"
   },
   "outputs": [],
   "source": [
    "# Поделим корпус на train и test\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(all_texts,\n",
    "                                                    lang_indices,\n",
    "                                                    test_size=0.2,\n",
    "                                                    random_state=0)\n",
    "clf.fit(X_train, y_train)\n",
    "y_predicted = clf.predict(X_test)\n",
    "cm = metrics.confusion_matrix(y_test, y_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 550,
     "status": "ok",
     "timestamp": 1568991372644,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "jJDrn7gp3Ipy",
    "outputId": "b1a85a3b-676b-460a-d4a2-057712f7242a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['be' 'be' 'uk' 'uk' 'uk' 'fr' 'uk' 'fr' 'kk' 'fr' 'uk' 'be' 'fr' 'uk'\n",
      " 'be' 'be' 'be' 'ru' 'be' 'kk' 'ru' 'ru' 'ru' 'kk' 'kk' 'kk' 'be' 'fr'\n",
      " 'uk' 'ru' 'fr' 'uk' 'fr' 'fr' 'kk' 'be' 'be' 'fr' 'ru' 'uk' 'ru' 'kk'\n",
      " 'be' 'kk' 'ru' 'kk' 'fr' 'be']\n",
      "['be', 'be', 'uk', 'uk', 'uk', 'fr', 'uk', 'fr', 'kk', 'fr', 'uk', 'be', 'fr', 'uk', 'be', 'be', 'be', 'uk', 'be', 'kk', 'ru', 'ru', 'ru', 'kk', 'kk', 'kk', 'be', 'kk', 'uk', 'uk', 'fr', 'uk', 'fr', 'fr', 'kk', 'be', 'be', 'fr', 'ru', 'uk', 'ru', 'kk', 'be', 'kk', 'ru', 'kk', 'fr', 'be']\n"
     ]
    }
   ],
   "source": [
    "print(y_predicted)\n",
    "print(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZQKO9WDx3Ipz"
   },
   "outputs": [],
   "source": [
    "# Нарисуем confusion matrix и оценим качество\n",
    "def test_classify(y_test, y_predicted, label_names):\n",
    "    cm = metrics.confusion_matrix(y_test, y_predicted)\n",
    "    \n",
    "    plt.figure(figsize=(10, 10))\n",
    "    plt.ylim(10.5, -0.5)\n",
    "    sns.heatmap(cm, annot=True,  fmt='', xticklabels=label_names, yticklabels=label_names)\n",
    "    plt.title('Confusion Matrix')\n",
    "    \n",
    "    print(metrics.classification_report(y_test, y_predicted,\n",
    "                                        target_names=label_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 811
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1071,
     "status": "ok",
     "timestamp": 1568991388247,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "GvUkK9oz3Ip0",
    "outputId": "b637536b-0c45-4787-c7b6-c33023106a51"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "          be       1.00      1.00      1.00        12\n",
      "          fr       0.90      1.00      0.95         9\n",
      "          kk       1.00      0.90      0.95        10\n",
      "          ru       0.75      1.00      0.86         6\n",
      "          uk       1.00      0.82      0.90        11\n",
      "\n",
      "    accuracy                           0.94        48\n",
      "   macro avg       0.93      0.94      0.93        48\n",
      "weighted avg       0.95      0.94      0.94        48\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAJOCAYAAACdoPzCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xu85XVdL/7Xey44QKAiKsxAISKG\nNzLES6VHK8EKlNID3sqS35lfhamZUp68o508v44nsYtNagheEjMzLiYc059iQiCSwoAiIDAzIKIp\niuJc9uf3x17w247DzJ7N/n7X+q55Pnmsx+zvd639Xe/1YT32/uzX57KqtRYAgEmyZNwFAABsTQcF\nAJg4OigAwMTRQQEAJo4OCgAwcXRQAICJo4MCPauq3avqrKr6dlV98B5c53lVdd5i1jYOVfXRqnrB\nuOsAJosOCtyNqnpuVV1SVd+tqptGv0h/bhEu/awkD0xyv9baf13oRVpr722tHbUI9fyQqnpyVbWq\n+vBW5w8fnf/kPK/zuqp6z44e11r7pdbauxdYLjCldFBgG6rqZUn+PMmfZLYz8eNJ/irJMxbh8j+R\n5Muttc2LcK2ufD3JE6rqfnPOvSDJlxfrCWqWn0HANvnhAFupqnsneUOSk1pr/9hau721tqm1dlZr\n7RWjx9yrqv68qjaMbn9eVfca3ffkqlpXVX9QVbeM0pffGt33+iSvSXLCKJk5ceukoaoOGiUVy0bH\nv1lV11bVd6rquqp63pzzF8z5vp+pqotHQ0cXV9XPzLnvk1V1SlV9ZnSd86pq3+00w8Yk/5Tk2aPv\nX5rkhCTv3aqt3lpVN1bVbVX1uap64uj805L89zmv8z/m1PGmqvpMku8lOXh07v8a3f/XVfWhOdd/\nc1V9vKpq3v8DgamggwI/6glJViT58HYe88dJHp/kp5IcnuSxSV415/79ktw7yaokJyb5y6q6b2vt\ntZlNZT7QWvux1to7t1dIVe2Z5NQkv9Ra2yvJzyS5bBuP2yfJOaPH3i/JW5Kcs1UC8twkv5XkAUl2\nS/Ly7T13ktOT/Mbo66OTXJ5kw1aPuTizbbBPkvcl+WBVrWit/ctWr/PwOd/z60lWJ9kryfVbXe8P\nkjxy1Pl6Ymbb7gXNZ3LALkcHBX7U/ZLcuoMhmOcleUNr7ZbW2teTvD6zv3jvtGl0/6bW2rlJvpvk\noQusZybJI6pq99baTa21K7bxmF9JcnVr7YzW2ubW2vuTXJXk2DmP+bvW2pdba99PcmZmOxZ3q7X2\nb0n2qaqHZrajcvo2HvOe1to3Rs/5v5LcKzt+nae11q4Yfc+mra73vcy241uSvCfJ77XW1u3gesAU\n0kGBH/WNJPveOcRyN1bmh//6v3507q5rbNXB+V6SH9vZQlprt2d2aOW3k9xUVedU1U/Oo547a1o1\n5/jmBdRzRpIXJXlKtpEoVdXLq+rK0bDStzKbGm1v6ChJbtzena21i5Jcm6Qy25ECdkE6KPCjPpvk\nB0mO285jNmR2suudfjw/OvwxX7cn2WPO8X5z72ytfay19tQk+2c2FfnbedRzZ03rF1jTnc5I8rtJ\nzh2lG3cZDcGcnOT4JPdtrd0nybcz27FIkrsbltnucE1VnZTZJGbD6PrALkgHBbbSWvt2Ziey/mVV\nHVdVe1TV8qr6par6n6OHvT/Jq6rq/qPJpq/J7JDEQlyW5ElV9eOjCbqvvPOOqnpgVT1jNBflB5kd\nKprZxjXOTXLoaGn0sqo6IcnDkpy9wJqSJK2165L8l8zOudnaXkk2Z3bFz7Kqek2Svefc/7UkB+3M\nSp2qOjTJG5M8P7NDPSdX1XaHooDppIMC2zCaT/GyzE58/XpmhyVelNmVLcnsL9FLknwhyReTXDo6\nt5DnOj/JB0bX+lx+uFOxZFTHhiTfzGxn4Xe2cY1vJDkms5NMv5HZ5OGY1tqtC6lpq2tf0FrbVjr0\nsST/ktmlx9cnuSM/PHxz5yZ036iqS3f0PKMhtfckeXNr7T9aa1dndiXQGXeukAJ2HWVyPAAwaSQo\nAMDE0UEBABZNVb1rtEnl5XPO/T9VdVVVfaGqPlxV99nRdXRQAIDFdFqSp2117vwkj2itPSqz89Ze\nufU3bU0HBQBYNK21T2V2Uv/cc+fN2RvqwiQH7Og629uIalFsuvVas3A7tvvKJ467BADm2Lxxfa+f\nH9Xn79rd7v/g/zuzH1dxpzWttTU7cYkXZnbl4nZ13kEBAKbHqDOyMx2Su1TVH2d2/6T37uixOigA\nQOeq6jczu1/TL8znA0B1UABg6Ga2jLuC7aqqp2V2A8n/svXHZtwdk2QBgEVTVe/P7GeaPbSq1lXV\niUn+IrMfj3F+VV1WVW/f0XUkKAAwdG1bH9E1Hq2152zj9Dt39joSFABg4khQAGDoZiYnQVksEhQA\nYOJIUABg4NoEzUFZLBIUAGDiSFAAYOjMQQEA6J4EBQCGzhwUAIDu6aAAABPHEA8ADN2Ef1jgQkhQ\nAICJI0EBgKEzSRYAoHsSFAAYOhu1AQB0T4ICAAPnwwIBAHogQQGAoTMHBQCgexIUABg6c1AAALon\nQQGAofNZPAAA3ZOgAMDQmYMCANA9HRQAYOIY4gGAobNRGwBA9yQoADB0JskCAHRPggIAQ2cOCgBA\n9yQoADBwrdnqHgCgcxIUABg6q3gAALonQQGAobOKBwCgexIUABg6c1AAALonQQGAoZuxDwoAQOd0\nUACAiWOIBwCGziTZ6fOqP3lLnvQrz85xz//tu8792V+8I8c+57/lV3/jd/LiV74ht33nu2OscPoc\nfdSTc8Xln8pVay/Iya84adzlTCVt3D1t3D1tvGvb5Tsox/3yU/P2t7zxh8494chH58NnvD0fPv2v\nc9CBq/KOMz4wpuqmz5IlS3LqW9+UY459fh55+FNywgnH5bDDHjLusqaKNu6eNu6eNt5JMzP93Xqy\ny3dQHvNTj8y9997rh8797OOOyLJlS5Mkj3r4T+Zrt9w6jtKm0mOPfHSuuearue66G7Jp06aceeZH\n8vRjjx53WVNFG3dPG3dPGzOvDkpVPbCq3llVHx0dP6yqTuy2tMnw4XPOy8894chxlzE1Vq7aLzeu\n23DX8br1N2Xlyv3GWNH00cbd08bd08Y7qc30d+vJfBOU05J8LMnK0fGXk7z07h5cVaur6pKquuQd\np7//nlU4Rn/z7vdn6dKlOeaop4y7FADYpcx3Fc++rbUzq+qVSdJa21xVd7srTGttTZI1SbLp1mvb\nPS+zf/90zvn51Gf+Pe849X+kqsZdztTYsP7mHHjAyruOD1i1fzZsuHmMFU0fbdw9bdw9bbyTduEP\nC7y9qu6XpCVJVT0+ybc7q2rMLrjwkrzrfR/M29782uy+YsW4y5kqF19yWQ455EE56KADs3z58hx/\n/DNy1tnnjbusqaKNu6eNu6eNmW+C8rIk/5zk4Kr6TJL7J3lWZ1X16BWv/dNc/Pkv5Fvfui2/cNzz\n87sn/nreccYHsnHTpvy3l/5xktmJsq89+ffGXOl02LJlS17y0lfl3HPel6VLluS0d38ga9d+edxl\nTRVt3D1t3D1tvJOmMEGp1nY8AlNVK5K8KMnRSb6T5LNJ3tZau2NH3zvUIZ4h2X3lE8ddAgBzbN64\nvte5AXd8+ozefteueOKv9/La5pugnJ7ktiR/Mjp+bpIzkvzXLooCAOavten7sMD5dlAe0Vp72Jzj\nT1TV2i4KAgCYbwfl0qp6fGvtwiSpqscluaS7sgCAeZvCOSjb7aBU1Rczu3JneZJ/q6obRsc/keSq\n7ssDAHZFO0pQjumlCgBg4abw04y320FprV3fVyEAAHfa5T8sEACYPPOdJAsATKopnCQrQQEAJo4E\nBQCGbgonyUpQAICJI0EBgKEzBwUAoHsSFAAYOnNQAAC6J0EBgKEzBwUAoHsSFAAYOgkKAED3JCgA\nMHRW8QAAdE+CAgBDZw4KAED3dFAAgIljiAcAhs4kWQCA7klQAGDoTJIFAOieDgoADF2b6e+2A1X1\nrqq6paoun3Nun6o6v6quHv173x1dRwcFAFhMpyV52lbn/ijJx1trD0ny8dHxdpmDAgBDN0FzUFpr\nn6qqg7Y6/YwkTx59/e4kn0zyh9u7jgQFAJi3qlpdVZfMua2ex7c9sLV20+jrm5M8cEffIEEBgKHr\nMUFpra1JsuYefH+rqrajx0lQAICufa2q9k+S0b+37OgbdFAAYOha6++2MP+c5AWjr1+Q5CM7+gYd\nFABg0VTV+5N8NslDq2pdVZ2Y5E+TPLWqrk7yi6Pj7TIHBQCGbrJW8Tznbu76hZ25jgQFAJg4EhQA\nGLoJSlAWiwQFAJg4EhQAGLp5fEbO0EhQAICJo4MCAEwcQzwAMHQmyQIAdE+CAgBDt/At6CeWBAUA\nmDgSFAAYuimcg9J5B2X3lU/s+il2ebedctS4S5h6e7/6vHGXALBLkaAAwNBNYYJiDgoAMHEkKAAw\ndLa6BwDongQFAAauzdgHBQCgcxIUABg6q3gAALonQQGAobOKBwCgezooAMDEMcQDAENnmTEAQPck\nKAAwdJYZAwB0T4ICAEMnQQEA6J4EBQCGrlnFAwDQOQkKAAydOSgAAN2ToADA0NlJFgCgexIUABi6\nZg4KAEDnJCgAMHTmoAAAdE8HBQCYOIZ4AGDgmo3aAAC6J0EBgKEzSRYAoHsSFAAYOhu1AQB0T4IC\nAENnDgoAQPckKAAwdPZBAQDongQFAIbOHBQAgO5JUABg6OyDAgDQPQkKAAydOSgAAN3TQQEAJo4h\nHgAYuGajNgCA7klQAGDoTJIFAOieBAUAhk6CMt2OPurJueLyT+WqtRfk5FecNO5yptayI56aFS88\nJStOfGOWPeap4y5nKnkvd08bd08b79p0UEaWLFmSU9/6phxz7PPzyMOfkhNOOC6HHfaQcZc1dWrf\nVVl2+JNyx+mn5I53vSZLH3x46j4PGHdZU8V7uXvauHvaeCe1mf5uPdFBGXnskY/ONdd8Ndddd0M2\nbdqUM8/8SJ5+7NHjLmvqLLnf/pm56dpk88akzWTLjV/K0kOPGHdZU8V7uXvauHvamO12UKpqaVW9\nt69ixmnlqv1y47oNdx2vW39TVq7cb4wVTaeZW9dnyQGHJiv2TJbtlqUHPyq19z7jLmuqeC93Txt3\nTxvvpJnW360n250k21rbUlU/UVW7tdY2zveiVbU6yeokqaX3zpIle97DMpkW7Rs3ZdNF52bFCS9P\n2/SDzNxyw1R+CicA98x8VvFcm+QzVfXPSW6/82Rr7S139w2ttTVJ1iTJst1WDWJq8Yb1N+fAA1be\ndXzAqv2zYcPNY6xoem35wqez5QufTpIsf9Iz077zzTFXNF28l7unjbunjXdO25VW8VTVGaMvn57k\n7NFj95pzmyoXX3JZDjnkQTnooAOzfPnyHH/8M3LW2eeNu6zptMfs26f22idLDz0im9deOOaCpov3\ncve0cfe0MdtLUI6oqpVJbkjytp7qGZstW7bkJS99Vc49531ZumRJTnv3B7J27ZfHXdZUutdxL0rt\nvmcysyUbzz8j+cH3x13SVPFe7p427p423klTmKBUa9t+UVX14iS/k+RBSTbMvStJa60dPJ8nGMoQ\nz5DddspR4y5h6u39an+5AfO3eeP66vP5vvPiY3r7XbvXqWf38truNkFprZ2a5NSq+uvW2u/0UQwA\nsAC74qcZ65wAAH2zURsAMHF8WCAADN0UTpKVoAAAE0eCAgBDJ0EBAOieBAUABu7u9jQbMgkKADBx\nJCgAMHTmoAAA3L2q+v2quqKqLq+q91fVioVcRwcFAIZupvV3246qWpXkxUke01p7RJKlSZ69kJek\ngwIALKZlSXavqmVJ9sgPf+DwTl0EABiw1uMclKpanWT1nFNrWmtrkqS1tr6q/izJDUm+n+S81tqC\nPg5eBwUAmLdRZ2TNtu6rqvsmeUaSByX5VpIPVtXzW2vv2dnnMcQDAEM3IXNQkvxikutaa19vrW1K\n8o9JfmYhL0kHBQBYLDckeXxV7VFVleQXkly5kAsZ4gGAoZsZdwGzWmsXVdU/JLk0yeYkn8/dDAft\niA4KALBoWmuvTfLae3odQzwAwMSRoADAwPW5zLgvEhQAYOJIUABg6CQoAADdk6AAwNBNyDLjxSRB\nAQAmjgQFAAbOKh4AgB5IUABg6MxBAQDongQFAAbOHBQAgB5IUABg6MxBAQDongQFAAauSVAAALqn\ngwIATBxDPAAwdIZ4AAC6J0EBgIEzSRYAoAcSFAAYOgkKAED3JCgAMHDmoAAA9ECCAgADJ0EBAOiB\nBAUABk6CAgDQAwnKFHj4/7x03CVMvdtOOWrcJewS9n71eeMuAYap1bgrWHQSFABg4khQAGDgzEEB\nAOiBDgoAMHEM8QDAwLUZk2QBADonQQGAgTNJFgCgBxIUABi4ZqM2AIDuSVAAYODMQQEA6IEEBQAG\nzj4oAAA9kKAAwMC1Nu4KFp8EBQCYOBIUABg4c1AAAHogQQGAgZOgAAD0QAcFAJg4hngAYOAsMwYA\n6IEEBQAGziRZAIAeSFAAYOBak6AAAHROggIAA9dmxl3B4pOgAAATR4ICAAM3Yw4KAED3JCgAMHBW\n8QAA9ECCAgADZydZAIAeSFAAYOB8mjEAQA90UACAiWOIBwAGziRZAIAeSFAAYOBsdQ8A0AMJCgAM\nnK3uAQB6IEEBgIGzURsAQA8kKAAwcFbxAAD0QIICAANnFQ8AQA90UOY4+qgn54rLP5Wr1l6Qk19x\n0rjLmUpvPvX1ufiqT+RfLvjQuEuZasuOeGpWvPCUrDjxjVn2mKeOu5yp5OdF97Tx/LXW360vOigj\nS5YsyalvfVOOOfb5eeThT8kJJxyXww57yLjLmjofev9H8pvH/864y5hqte+qLDv8Sbnj9FNyx7te\nk6UPPjx1nweMu6yp4udF97TxcFXVfarqH6rqqqq6sqqesJDrzKuDUlX7bOPcgxbyhJPqsUc+Otdc\n89Vcd90N2bRpU8488yN5+rFHj7usqfPvn7003/rP28ZdxlRbcr/9M3PTtcnmjUmbyZYbv5Slhx4x\n7rKmip8X3dPGO2emVW+3eXhrkn9prf1kksOTXLmQ1zTfBOWsqtr7zoOqeliSsxbyhJNq5ar9cuO6\nDXcdr1t/U1au3G+MFcHCzNy6PksOODRZsWeybLcsPfhRqb1/5G8M7gE/L7qnjYepqu6d5ElJ3pkk\nrbWNrbVvLeRa813F8yeZ7aT8SpKHJjk9yfO2U+DqJKuTpJbeO0uW7LmQ2oAFaN+4KZsuOjcrTnh5\n2qYfZOaWG5I2M+6ygA71uYpn7u/4kTWttTWjrx+U5OtJ/q6qDk/yuSQvaa3dvrPPM68OSmvtnKpa\nnuS8JHsl+dXW2pe38/g1SdYkybLdVg1iA94N62/OgQesvOv4gFX7Z8OGm8dYESzcli98Olu+8Okk\nyfInPTPtO98cc0XTxc+L7mnjyTX3d/w2LEvy00l+r7V2UVW9NckfJXn1zj7Pdod4quptVXVqVZ2a\n5BeS3DvJdUleNDo3NS6+5LIccsiDctBBB2b58uU5/vhn5Kyzzxt3WbAwe+yVJKm99snSQ4/I5rUX\njrmg6eLnRfe08WCtS7KutXbR6PgfMtth2Wk7SlAuGf27b5Jb5xwns0nK1NiyZUte8tJX5dxz3pel\nS5bktHd/IGvX3m1IxAK9dc2f5vE/+5jc9373yb998bz8+Z/+dc5874fHXdbUuddxL0rtvmcysyUb\nzz8j+cH3x13SVPHzonvaeOdMylb3rbWbq+rGqnpoa+1LmQ031i7kWtXmsai5qi5N8oLW2hdHx89J\n8tLW2uN29L1DGeIZsgP32nfcJUy9K05e0B8A7KS9X+0vZKbD5o3re+0xXLTy13r7Xfu4Df+43ddW\nVT+V5B1JdktybZLfaq39584+z3wnyT4ryT9U1XOTPDHJbyQ5amefDABYfJOUBLTWLkvymHt6nflO\nkr22qp6d5J+S3JDkqNaazBgA6MR2OyhV9cX8cMdsnyRLk1xUVWmtParL4gCAHZuUOSiLaUcJyjG9\nVAEAMMd2Oyittev7KgQAWJg+N2rriw8LBAAmznxX8QAAE2oaP8xCggIATBwJCgAMXIs5KAAAnZOg\nAMDAzUzSVrKLRIICAEwcCQoADNyMOSgAAN3TQQEAJo4hHgAYOMuMAQB6IEEBgIGz1T0AQA8kKAAw\ncOagAAD0QIICAANnDgoAQA8kKAAwcBIUAIAeSFAAYOCs4gEA6IEEBQAGbmb6AhQJCgAweSQoADBw\nM+agAAB0TwcFAJg4hngAYODauAvogAQFAJg4EhQAGDhb3QMA9ECCAgADN1OWGQMAdE6CAgADZxUP\nAEAPJCgAMHBW8QAA9ECCAgADNzN9i3gkKADA5JGgAMDAzWT6IhQJCgAwcSQoADBw9kEBAOiBDgoA\nMHEM8UyBG79z67hLmHp7v/q8cZewS7jwAUeOu4Sp9/hbLh53CXTAMmMAgB5IUABg4Gx1DwDQAwkK\nAAycZcYAAD2QoADAwFnFAwDQAwkKAAycVTwAAD2QoADAwElQAAB6IEEBgIFrVvEAAHRPggIAA2cO\nCgBAD3RQAICJY4gHAAbOEA8AQA8kKAAwcG3cBXRAggIATBwJCgAM3IyN2gAAuidBAYCBs4oHAKAH\nEhQAGDgJCgBADyQoADBw9kEBAOiBBAUABs4+KAAAPZCgAMDAWcUDANADHRQAYOLooADAwLUeb/NR\nVUur6vNVdfZCX5MOCgCw2F6S5Mp7cgEdFAAYuJm03m47UlUHJPmVJO+4J69JBwUAmLeqWl1Vl8y5\nrd7qIX+e5OTcw8VFlhkDwMD1ucy4tbYmyZpt3VdVxyS5pbX2uap68j15HgkKALBYfjbJ06vqq0n+\nPsnPV9V7FnIhHRQAGLhJWcXTWntla+2A1tpBSZ6d5F9ba89fyGvSQQEAJo45KAAwcJO41X1r7ZNJ\nPrnQ75egAAATR4ICAAM3U+OuYPFJUACAiSNBAYCBm88Or0MjQQEAJo4EBQAGbvryEwkKADCBdFAA\ngImjgzLH0Uc9OVdc/qlctfaCnPyKk8ZdztTSzt3Txt1buveeOfhvTs7DP/kXefgn3pY9f/qh4y5p\n6ngfz99Mj7e+6KCMLFmyJKe+9U055tjn55GHPyUnnHBcDjvsIeMua+po5+5p434c+PoTc9snL80V\nT35R1h71+7njK+vGXdJU8T5GB2XksUc+Otdc89Vcd90N2bRpU8488yN5+rFHj7usqaOdu6eNu7d0\nrz2y1+Menlvf/3+SJG3T5my57fYxVzVdvI93zkxab7e+zGsVT1X9XbYxSbi19sJFr2hMVq7aLzeu\n23DX8br1N+WxRz56jBVNJ+3cPW3cvd0OfGA2f/PbOegtL84eDzsot3/xmtz4mndk5vs/GHdpU8P7\nmPkmKGcnOWd0+3iSvZN89+4eXFWrq+qSqrpkZsZfFcB0qWVLsscjHpyvn/HRrH3ayzLzvTuy30nP\nHHdZ7MJaj7e+zCtBaa19aO5xVb0/yQXbefyaJGuSZNluqwaxPHvD+ptz4AEr7zo+YNX+2bDh5jFW\nNJ20c/e0cfc23vSNbLzpG7n981cnSf7znM9mv5N+bcxVTRfvYxY6B+UhSR6wmIWM28WXXJZDDnlQ\nDjrowCxfvjzHH/+MnHX2eeMua+po5+5p4+5t/vq3snHDrbnXwbO/QPf+uUfljqtvHHNV08X7eOdM\n4yqeHSYoVVVJtuSHh3RuTvKHXRU1Dlu2bMlLXvqqnHvO+7J0yZKc9u4PZO3aL4+7rKmjnbunjftx\nw6v/Nge/7WWp3ZblB9d/LV/9g1PHXdJU8T6mWtvxCExVXd5ae8RCnmAoQzzA+F34gCPHXcLUe/wt\nF4+7hF3C5o3rq8/ne9lBz+7td+1bvvr3vby2+Q7xfK6q/OQAAHox3w8LfFyS51XV9UluT1JJWmvt\nUZ1VBgDMyzQOVcy3g2J3HACgN/NdZnx914UAAAvT5+qavtjqHgCYOPMd4gEAJlSbwlkoEhQAYOLo\noAAAE8cQDwAMnEmyAAA9kKAAwMDNmCQLANA9CQoADNz05ScSFABgAklQAGDgzEEBAOiBBAUABs4+\nKAAAPZCgAMDA+bBAAIAeSFAAYODMQQEA6IEEBQAGzhwUAIAe6KAAABPHEA8ADJxJsgAAPZCgAMDA\nzTSTZAEAOidBAYCBm778RIICAEwgCQoADNzMFGYoEhQAYOJIUABg4Gx1DwDQAwkKAAycnWQBAHog\nQQGAgbOKBwCgBxIUABg4q3gAAHqggwIATBxDPAAwcJYZAwD0QIICAAPXmkmyAACdk6AAwMDZqA0A\noAcSFAAYuGlcxaODAkyMl+e74y5h6t12ylHjLgHmRQcFAAbOVvcAAD2QoADAwFnFAwDQAwkKAAyc\nnWQBAHogQQGAgZvGfVAkKADAxJGgAMDA2QcFAKAHOigAwMQxxAMAA2ejNgCAHkhQAGDgbNQGAHA3\nqurAqvpEVa2tqiuq6iULvZYEBQAGboLmoGxO8gettUuraq8kn6uq81tra3f2QhIUAGBRtNZuaq1d\nOvr6O0muTLJqIdeSoADAwPW5UVtVrU6yes6pNa21Ndt43EFJHp3kooU8jw4KADBvo87Ij3RI5qqq\nH0vyoSQvba3dtpDn0UEBgIGbmaBVPFW1PLOdk/e21v5xodcxBwUAWBRVVUnemeTK1tpb7sm1dFAA\nYOBaj7cd+Nkkv57k56vqstHtlxfymgzxAACLorV2QZJajGvpoADAwE3QPiiLxhAPADBxJCgAMHAS\nFACAHuigAAATxxAPAAxcm6CN2haLBAUAmDgSFAAYOJNkAQB6IEEBgIFrEhQAgO5JUABg4KziAQDo\ngQQFAAbOKh4AgB5IUABg4MxBAQDogQQFAAbOHBQAgB5IUABg4OwkCwDQAx0UAGDiGOIBgIGbscwY\nAKB7EhQAGDiTZAEAeqCDMsfRRz05V1z+qVy19oKc/IqTxl3O1NLO3dPG3br//vfP/z7zz3Lav74z\nf/fxd+SZJ/7quEuaSsuOeGpWvPCUrDjxjVn2mKeOu5yJNtNab7e+GOIZWbJkSU5965vytF9+Ttat\nuykXfvbcnHX2ebnyyqvHXdrW3i9+AAAH80lEQVRU0c7d08bd27JlS/7qDW/P1Zd/JbvvuXvWfPSv\nc8mnPpfrr75h3KVNjdp3VZYd/qTccfopyZbNudfxL8uWr/xH2rduGXdp9ESCMvLYIx+da675aq67\n7oZs2rQpZ575kTz92KPHXdbU0c7d08bd++Yt38zVl38lSfL927+f66++Ifvut++Yq5ouS+63f2Zu\nujbZvDFpM9ly45ey9NAjxl3WxGo9/tcXHZSRlav2y43rNtx1vG79TVm5cr8xVjSdtHP3tHG/9jvg\ngXnIIw7JlZ+/atylTJWZW9dnyQGHJiv2TJbtlqUHPyq19z7jLosezWuIp6re0Fp7zZzjpUlOb609\n724evzrJ6iSppffOkiV7LkatABNl9z1W5PVrXpu/eN1f5Xvf/d64y5kq7Rs3ZdNF52bFCS9P2/SD\nzNxyQ9Jmxl3WxJrGfVDmOwflwKp6ZWvtf1TVvZKcmeTzd/fg1tqaJGuSZNluqwbRahvW35wDD1h5\n1/EBq/bPhg03j7Gi6aSdu6eN+7F02dK8fs3r8n8+/PF8+qMXjLucqbTlC5/Oli98Okmy/EnPTPvO\nN8dcEX2a7xDPC5M8sqpemeSsJJ9orb2us6rG4OJLLsshhzwoBx10YJYvX57jj39Gzjr7vHGXNXW0\nc/e0cT9O/rOX54avXJ8P/u2Hxl3K9NpjryRJ7bVPlh56RDavvXDMBU2uaZyDst0Epap+es7hW5P8\nTZLPJPlUVf10a+3SLovr05YtW/KSl74q557zvixdsiSnvfsDWbv2y+Mua+po5+5p4+498shH5Ohn\nPTXXXHlt3vGxtydJ/vbN78pF//rvY65sutzruBeldt8zmdmSjeefkfzg++MuiR5V2864VVV9YvTl\nnQ+qOV+ntfbzO3qCoQzxAOP3cw84bNwlTL1zf+/AcZewS9jjD/+u+ny+B+/70739rr3m1kt7eW3b\nTVBaa09Jkqp67bbu7qQiAGCXN99Jst+d8/WKJMckuXLxywEAdtY0fhbPvDoorbX/Nfe4qv4sycc6\nqQgA2OUtdKO2PZIcsJiFAADcab4btX0x//+ck6VJ7p/kDV0VBQDMX5vCTezmOwflmDlfb07ytdba\n5g7qAQCY9xyU67suBABYmJkpnCTrwwIBgIkz3yEeAGBCbW/T1aGSoAAAE0eCAgADZw4KAEAPJCgA\nMHDmoAAA9ECCAgADNyNBAQDongQFAAauWcUDANA9CQoADJxVPAAAPdBBAQAmjiEeABg4W90DAPRA\nggIAA2eSLABADyQoADBwtroHAOiBBAUABs4cFACAHkhQAGDg7IMCANADCQoADJw5KAAAPZCgAMDA\n2QcFAKAHEhQAGLhmFQ8AQPd0UACAiWOIBwAGziRZAIAeSFAAYOBs1AYA0AMJCgAMnGXGAAA9kKAA\nwMCZgwIA0AMJCgAMnAQFAGA7quppVfWlqvpKVf3RQq+jgwIAA9d6vG1PVS1N8pdJfinJw5I8p6oe\ntpDXpIMCACyWxyb5Smvt2tbaxiR/n+QZC7lQ53NQNm9cX10/x2KrqtWttTXjrmOaaePuaeN+aOfu\naeMd6/N3bVWtTrJ6zqk1c/7/rEpy45z71iV53EKeR4Kybat3/BDuIW3cPW3cD+3cPW08QVpra1pr\nj5lz66TzqIMCACyW9UkOnHN8wOjcTtNBAQAWy8VJHlJVD6qq3ZI8O8k/L+RC9kHZNmOd3dPG3dPG\n/dDO3dPGA9Fa21xVL0rysSRLk7yrtXbFQq5V07i5CwAwbIZ4AICJo4MCAEycXb6DUlUHVdXl465j\nV1FVL66qK6vqveOuZdps671cVb9ZVX8xrppgsVTVJ6vqMeOug/6YJEvffjfJL7bW1t15oqqWtdY2\nj7Em2GlVVZmdxzcz7lpgGu3yCcrIsqp67+gv+3+oqj2q6oiq+n+r6nNV9bGq2n/cRQ5dVb09ycFJ\nPlpV366qM6rqM0nOGHNpU6eqDq6qzye5/5xzv1JVn62qfcdY2qCNUqovVdXpSS5PsmXOfc+qqtPG\nVtyU2DoJrKqXV9Xr5hwvqarTquqNYymQ3uigzHpokr9qrR2W5LYkJyV5W5JntdaOSPKuJG8aY31T\nobX220k2JHlKkv+d2Q+S+sXW2nPGWtiUqaqHJvlQkt9M8vXRuV9N8kdJfrm1duv4qpsKD8nsz4uH\nJ7l93MXsYpYleW+Sq1trrxp3MXTLEM+sG1trnxl9/Z4k/z3JI5KcP5viZmmSm8ZU2zT759ba98dd\nxJS5f5KPJPm11traqnp0kp9P8pgkR7XWbhtrddPh+tbaheMuYhf1N0nObK35g3EXIEGZtfVmMN9J\nckVr7adGt0e21o4aR2FTzl+fi+/bSW5I8nNzzl2TZK8kh46loukz930792fHir4LmVKb88O/m+a2\n678leUpVaetdgA7KrB+vqieMvn5ukguT3P/Oc1W1vKoePrbqYP42JvnVJL9RVc8dnbs+yTOTnO59\nvOi+VlWHVdWSzLY799zXkjygqu5XVfdKcsyc+96Z5NwkZ1aVEYApp4My60tJTqqqK5PcN6P5J0ne\nXFX/keSyJD8zxvpg3lprt2f2h/rvJ9l7dO6qJM9L8sGqevAYy5s2f5Tk7Mz+ZW8YeBG01jYleUOS\nf09yfpKrtrr/LUk+n+SMUceQKWWrewBg4uh9AgATRwcFAJg4OigAwMTRQQEAJo4OCgAwcXRQAICJ\no4MCAEyc/w9FAkxDFgzwuwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_classify(y_test, y_predicted, clf.classes_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 811
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 840,
     "status": "ok",
     "timestamp": 1568991497246,
     "user": {
      "displayName": "Ekaterina Enikeeva",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDpPi2nAYgMUpBsdQCKgNbc8_vU2k6n8XpypQKMzA=s64",
      "userId": "10839941398806369136"
     },
     "user_tz": -180
    },
    "id": "0VjSFWhX3Ip1",
    "outputId": "2503d518-8aca-4343-e539-75d0cb36438c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "          be       1.00      0.50      0.67        12\n",
      "          fr       0.75      1.00      0.86         9\n",
      "          kk       0.83      0.50      0.62        10\n",
      "          ru       0.27      1.00      0.43         6\n",
      "          uk       0.50      0.09      0.15        11\n",
      "\n",
      "    accuracy                           0.56        48\n",
      "   macro avg       0.67      0.62      0.55        48\n",
      "weighted avg       0.71      0.56      0.55        48\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAisAAAJOCAYAAAB2l0fBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XuYXHWd5/H3tzuBhLtcNOQiAcEs\nICsioLLrDDojKAZwRuQi7njbjSjOCCqOFx50UPRx2HEQQTEryh2JNyQhKDMsgqIgAYOScL8nnQAG\nSASCJN3f/aMqbNEm3RXo6l+dk/fLpx7r1Dl16tuH86R//fldKjITSZKkbtVTugBJkqSh2FiRJEld\nzcaKJEnqajZWJElSV7OxIkmSupqNFUmS1NVsrEijLCLGR8TsiFgeET94Eec5OiKuHMnaSoiIKyLi\nvaXrkNS9bKxI6xAR746IeRHxZEQsaf5S/e8jcOrDgJcB22Tmu17oSTLzwsw8YATqeZ6I2D8iMiJ+\nMuj1Vzdf/0Wb5/lCRFww3HGZ+bbMPPcFlitpA2BjRVqLiPg4cBrwZRoNi5cD3wQOHYHT7wDcmZmr\nR+BcnfIo8IaI2KbltfcCd47UB0SD/wZJGpb/UEiDRMSWwMnAsZn548x8KjNXZebszDyheczGEXFa\nRPQ1H6dFxMbNfftHxKKI+EREPNJMZd7f3PcvwEnAEc3E5oODE4iImNpMMMY0t98XEfdGxJ8i4r6I\nOLrl9V+1vG+/iLix2b10Y0Ts17LvFxHxxYi4rnmeKyNi2yEuw7PApcCRzff3AkcAFw66Vl+PiIci\nYkVE3BQRb2y+/lbgsy0/5y0tdZwSEdcBTwM7NV/7n83934qIH7Wc/6sRcVVERNv/ASXVjo0V6S+9\nARgH/GSIYz4HvB7YE3g1sC9wYsv+CcCWwCTgg8CZEfGSzPw8jbTmkszcLDPPHqqQiNgUOB14W2Zu\nDuwHzF/LcVsDlzeP3Qb4GnD5oGTk3cD7gZcCGwGfHOqzgfOAf2g+PxC4FegbdMyNNK7B1sBFwA8i\nYlxm/mzQz/nqlvf8D2AGsDnwwKDzfQLYo9kQeyONa/fe9HtBpA2ajRXpL20D/HGYbpqjgZMz85HM\nfBT4Fxq/hNdY1dy/KjPnAk8C015gPQPAqyJifGYuycwFaznm7cBdmXl+Zq7OzIuB24GDW475Xmbe\nmZkrgVk0GhnrlJm/BraOiGk0Gi3nreWYCzJzWfMz/w3YmOF/znMyc0HzPasGne9pGtfxa8AFwD9m\n5qJhziep5mysSH9pGbDtmm6YdZjI81OBB5qvPXeOQY2dp4HN1reQzHyKRvfLMcCSiLg8Iv5LG/Ws\nqWlSy/bSF1DP+cBHgTexlqQpIj4ZEbc1u56eoJEmDdW9BPDQUDsz8wbgXiBoNKokbeBsrEh/6TfA\nn4F3DHFMH42Bsmu8nL/sImnXU8AmLdsTWndm5s8z8y3A9jTSkv/TRj1ralr8Amta43zgI8DcZurx\nnGY3zaeAw4GXZOZWwHIajQyAdXXdDNmlExHH0kho+prnl7SBs7EiDZKZy2kMgj0zIt4REZtExNiI\neFtE/GvzsIuBEyNiu+ZA1ZNodFu8EPOBv4qIlzcH935mzY6IeFlEHNocu/JnGt1JA2s5x1zglc3p\n1mMi4ghgN2DOC6wJgMy8D/hrGmN0BtscWE1j5tCYiDgJ2KJl/8PA1PWZ8RMRrwS+BLyHRnfQpyJi\nyO4qSfVnY0Vai+b4i4/TGDT7KI2ui4/SmCEDjV+o84DfA38Abm6+9kI+6z+AS5rnuonnNzB6mnX0\nAY/RaDh8eC3nWAZMpzFAdRmNRGJ6Zv7xhdQ06Ny/ysy1pUY/B35GYzrzA8AzPL+LZ82Cd8si4ubh\nPqfZ7XYB8NXMvCUz76Ixo+j8NTOtJG2YwkH2kiSpm5msSJKkrmZjRZIkjZiI+FhE3BoRCyLiuLXs\nj4g4PSLujojfR8Rew53TxookSRoREfEq4H/RWCjz1cD0iNh50GFvA3ZpPmYA3xruvDZWJEnSSNkV\nuCEzn26uNXUN8PeDjjkUOC8brge2iojthzrpUItejYjl7/9bR/B22H5z/1S6hNq743EXUR0Nd07b\nvXQJtffKO9a2ALJG2upnF4/q91mt+uO9o/a7dqPtXvEhGonIGjMzc2bz+a3AKc2v+lgJHERj5mSr\nSTx/5uCi5mtL1vWZHW+sSJKk+mg2TGauY99tEfFV4EoaC17OB/pf7GfaDSRJkkZMZp6dma/NzL8C\nHqexFlOrxcCUlu3JDLPatsmKJElVN/Ciw4sRExEvzcxHIuLlNMarvH7QIZcBH42I7wOvA5Zn5jq7\ngMDGiiRJGlk/ao5ZWQUcm5lPRMQxAJl5Fo2vBzkIuJvGl6q+f7gT2liRJKnqcm1fGVZGZr5xLa+d\n1fI8gWPX55yOWZEkSV3NZEWSpKob6J5kpRNMViRJUlczWZEkqeKyi8asdILJiiRJ6momK5IkVZ1j\nViRJksoxWZEkqeocsyJJklSOjRVJktTV7AaSJKnquuiLDDvBZEWSJHU1kxVJkqrOAbaSJEnlmKxI\nklR1LgonSZJUjsmKJEkV5xcZSpIkFWSyIklS1TlmRZIkqRyTFUmSqs4xK5IkSeWYrEiSVHV+N5Ak\nSVI5JiuSJFWdY1YkSZLKsbEiSZK6mt1AkiRVnYvCSZIklWOyIklS1TnAVpIkqRyTFUmSqs4xK5Ik\nSeWYrEiSVHGZLrcvSZJUjMmKJElV52wgSZKkckxWJEmqOmcDSZIklWOyIklS1TlmRZIkqRyTFUmS\nqm7AdVYkSZKKsbEiSZK6mt1AkiRVnQNsNyDjN2WTj5zEZl/+Lpudcja9r9i1dEW188XTTuTaBVdw\n6TUXlS6l1g48YH8W3Hotty/8FZ864djS5dTSlJ+dx+Qff5tJP/gWk75/Rulyasn7WGuYrLQYf/Sx\nrLr1RlZ982ToHQMbbVy6pNq59PtzuOjsH/CVMz5fupTa6unp4fSvn8JbDzqKRYuWcP1v5jJ7zpXc\ndttdpUurnb4PnMDAEytKl1FL3sfryUXhNhDjN2XMK/dg1bVXNLb7V8PKp8rWVEM3XT+f5f7j3lH7\n7vMa7rnnfu6770FWrVrFrFk/5ZCDDyxdlrRevI/Vqq3GSkS8LCLOjogrmtu7RcQHO1va6OrZdgID\nf1rO+A+ewGZfOIvx7/84bDSudFnSeps4aQIPLep7bnvR4iVMnDihYEU1lbD9t7/CpEvOZPPDDipd\nTe14H6+nHBi9RwHtJivnAD8HJja37wSOW9fBETEjIuZFxLxz7lj84iocJdHbS+8Ou/Ds1bN58gvH\nkH9+ho3ffmTpsiR1qb73Hs/iI45lyYc/xxZHHsy41+5RuiSpttptrGybmbOAAYDMXA2scwWazJyZ\nmXtn5t7vmzZpBMrsvIHHHiUff5T+e28HYNWN19K7wy6Fq5LWX9/ipUyZPPG57cmTtqevb2nBiuqp\n/5FlAAw89gRPX/VrNn7VtMIV1Yv38XoaGBi9RwHtNlaeiohtgASIiNcDyztWVQG54nEGHnuUngmT\nARiz214M9D1QuCpp/d04bz4777wjU6dOYezYsRx++KHMnnNl6bJqJcaPIzYZ/9zz8fvtxbN331+2\nqJrxPlardmcDfRy4DNgpIq4DtgMO61hVhay84AzGz/gMMWYsA48u4emzTy1dUu2cetYX2We/vdhq\n66246nezOfPUmfz4otmly6qV/v5+Pnbcicy9/CJ6e3o459xLWLjwztJl1UrvNlvxstMaM9qit5cn\n517NyuvmFa6qXryP11PNZwNFZg5/UMQ44KPAgcCfgN8A38jMZ4Z77/L3/+3wH6AXZb+5fypdQu3d\n8fii0iVsEO6ctnvpEmrvlXcsKF3CBmH1s4tjND/vmV+eP2q/a8e98X+M6s8G7Scr5wErgC83t98N\nnA+8qxNFSZKk9mXW+4sM222svCozd2vZvjoiFnaiIEmSpFbtNlZujojXZ+b1ABHxOsAOWkmSukHN\nx6wM2ViJiD/QmAE0Fvh1RDzY3N4BuL3z5UmSpA3dcMnK9FGpQpIkvXA1/9blIRsrmelCI5IkqSi/\nyFCSJHW1dgfYSpKkblXzAbYmK5IkqauZrEiSVHU1H2BrsiJJkrqayYokSVXnmBVJkqRyTFYkSao6\nx6xIkiSVY7IiSVLVOWZFkiSpHJMVSZKqzmRFkiSpHJMVSZKqztlAkiRJ5ZisSJJUdY5ZkSRJKsfG\niiRJ6mp2A0mSVHUOsJUkSSrHZEWSpKpzgK0kSVI5NlYkSaq6HBi9xxAiYlpEzG95rIiI4wYds39E\nLG855qThfjy7gSRJ0ojIzDuAPQEiohdYDPxkLYf+MjOnt3teGyuSJFVdd45Z+Rvgnsx84MWeyG4g\nSZLUtoiYERHzWh4z1nHokcDF69j3hoi4JSKuiIjdh/tMkxVJkqpuFJOVzJwJzBzqmIjYCDgE+Mxa\ndt8M7JCZT0bEQcClwC5Dnc9kRZIkjbS3ATdn5sODd2Tmisx8svl8LjA2IrYd6mQmK5IkVV1m6QoG\nO4p1dAFFxATg4czMiNiXRnCybKiT2ViRJEkjJiI2Bd4CfKjltWMAMvMs4DDgwxGxGlgJHJk5dGvL\nxookSVXXRbOBMvMpYJtBr53V8vwM4Iz1OadjViRJUlczWZEkqeq6KFnpBJMVSZLU1UxWJEmqumG+\ns6fqTFYkSVJXs7EiSZK6mt1AkiRVnQNsJUmSyjFZkSSp6rpvuf0RZbIiSZK6msmKJElVV/MxKx1v\nrGxz4W2d/ogN3sq+X5YuofbGT3xj6RI2CK+8Y0HpEiR1IZMVSZKqrubJimNWJElSVzNZkSSp6lxu\nX5IkqRyTFUmSKi4HXGdFkiSpGJMVSZKqztlAkiRJ5ZisSJJUdc4GkiRJKsfGiiRJ6mp2A0mSVHVO\nXZYkSSrHZEWSpKpz6rIkSVI5JiuSJFWdyYokSVI5JiuSJFVdOhtIkiSpGJMVSZKqzjErkiRJ5Zis\nSJJUda5gK0mSVI7JiiRJVZeOWZEkSSrGZEWSpKpzzIokSVI5NlYkSVJXsxtIkqSKSxeFkyRJKsdk\nRZKkqnOArSRJUjkmK5IkVZ2LwkmSJJVjsiJJUtU5ZkWSJKkckxVJkqrOdVYkSZLKMVmRJKnqHLMi\nSZJUjsmKJElV5zorkiRJ5ZisSJJUdY5ZkSRJKsfGiiRJ6mp2A0mSVHHponCSJEnlmKxIklR1DrCV\nJEkqx2RFkqSqM1nZcBx4wP4suPVabl/4Kz51wrGly6mt82ddyjvecwyHHv0hzr/kJ6XLqSXv5c7z\nGnee11hr2Fhp6unp4fSvn8L0g9/DHq9+E0cc8Q523XWX0mXVzl333s+PLvsZF3/nNH507je55te/\n5cFFfaXLqhXv5c7zGnee13g95cDoPQqwsdK07z6v4Z577ue++x5k1apVzJr1Uw45+MDSZdXOvfc/\nxB67T2P8uHGMGdPL3nvuwX9ec13psmrFe7nzvMad5zVWqyEbKxHRGxEXjlYxJU2cNIGHWv7CX7R4\nCRMnTihYUT3tvNMO3HzLAp5YvoKVzzzDL39zI0sffrR0WbXivdx5XuPO8xqvp4EcvUcBQw6wzcz+\niNghIjbKzGfbPWlEzABmAETvlvT0bPoiy1RdvGLqy/nA0e9ixvGfY/y4cUzbZSd6egz4JEnr1s5s\noHuB6yLiMuCpNS9m5tfW9YbMnAnMBBiz0aRKDFHuW7yUKZMnPrc9edL29PUtLVhRfb3z4AN5ZzPO\nPe2sc5jw0m0LV1Qv3sud5zXuPK/x+skNdTZQRJzffHoIMKd57OYtj1q5cd58dt55R6ZOncLYsWM5\n/PBDmT3nytJl1dKyx58AYMnSR7jqmus46C37ly2oZryXO89r3HleY7UaKll5bURMBB4EvjFK9RTT\n39/Px447kbmXX0RvTw/nnHsJCxfeWbqsWjr+s1/iiRUrGDNmDJ/7xEfYYvPNSpdUK97Lnec17jyv\n8XqqebISmWv/ASPin4APAzsCrXNLA8jM3KmdD6hKN1CVrez7ZekSam/8xDeWLkFShax+dnGM5uf9\n6Z+mj9rv2s1PnzOqPxsMkaxk5unA6RHxrcz88CjWJEmS1seG/q3LNlQkSVJJzhmVJEldzS8ylCSp\n6mo+wNZkRZIkdTWTFUmSqs5kRZIkqRyTFUmSKm5da6bVhcmKJEnqaiYrkiRVnWNWJEmS2hMRW0XE\nDyPi9oi4LSLeMGh/RMTpEXF3RPw+IvYa7pwmK5IkVV13JStfB36WmYdFxEbAJoP2vw3Ypfl4HfCt\n5v+vk8mKJEkaERGxJfBXwNkAmflsZj4x6LBDgfOy4Xpgq4jYfqjz2liRJKniciBH7RERMyJiXstj\nRkspOwKPAt+LiN9FxHciYtNB5U4CHmrZXtR8bZ1srEiSpLZl5szM3LvlMbNl9xhgL+Bbmfka4Cng\n0y/2Mx2zIklS1XXPmJVFwKLMvKG5/UP+srGyGJjSsj25+do6maxIkqQRkZlLgYciYlrzpb8BFg46\n7DLgH5qzgl4PLM/MJUOd12RFkqSqGyhdwPP8I3BhcybQvcD7I+IYgMw8C5gLHATcDTwNvH+4E9pY\nkSRJIyYz5wN7D3r5rJb9CRy7Pue0G0iSJHU1kxVJkiouu2eAbUeYrEiSpK5msiJJUtWZrEiSJJVj\nsiJJUtV119TlEWeyIkmSuprJiiRJFedsIEmSpIJMViRJqjrHrEiSJJVjsiJJUsU5ZkWSJKkgkxVJ\nkqrOMSuSJEnlmKxIklRxabIiSZJUjo0VSZLU1ewGkiSp6uwGkiRJKsdkRZKkinOArSRJUkEmK5Ik\nVZ3JiiRJUjkmK5IkVZxjViRJkgoyWZEkqeJMViRJkgoyWZEkqeJMViRJkgoyWamBd7/2+NIl1N6d\n03YvXcIG4Z9XjC9dQu0tfGZp6RLUCRmlK+gokxVJktTVTFYkSao4x6xIkiQVZGNFkiR1NbuBJEmq\nuBxwgK0kSVIxJiuSJFWcA2wlSZIKMlmRJKni0kXhJEmSyjFZkSSp4hyzIkmSVJDJiiRJFec6K5Ik\nSQWZrEiSVHGZpSvoLJMVSZLU1UxWJEmqOMesSJIkFWSyIklSxZmsSJIkFWRjRZIkdTW7gSRJqjin\nLkuSJBVksiJJUsU5wFaSJKkgkxVJkiou02RFkiSpGJMVSZIqLgdKV9BZJiuSJKmrmaxIklRxA45Z\nkSRJKsdkRZKkinM2kCRJUkEmK5IkVZwr2EqSJBVksiJJUsX5rcuSJEkF2ViRJEldzW4gSZIqzgG2\nkiRJBZmsSJJUcS63L0mSVJDJiiRJFedy+5IkSQWZrEiSVHEuCidJklSQyYokSRXnbCBJkqSCTFYk\nSao4ZwNJkiQVZLLS4sAD9udrXzuZ3p4evvu9i/nXU88sXVLtbLP9tnz0349jq223IjP5z4t+ztzv\nzSldVu1M+dl55NMryf4B6O9n8ZEfLV1SrXgfj44vnnYif/2W/8Zjf3ycd/z1u0uX09XqPhvIxkpT\nT08Pp3/9FN560FEsWrSE638zl9lzruS22+4qXVqt9Pf3c96Xvst9t97LuE3H89U5/8bvf3ULi+56\nqHRptdP3gRMYeGJF6TJqyft4dFz6/TlcdPYP+MoZny9ditZTRPQC84DFmTl90L73AacCi5svnZGZ\n3xnqfG11A0XE1mt5bcd23lsV++7zGu65537uu+9BVq1axaxZP+WQgw8sXVbtPPHI49x3670APPPU\nShbfvYitX/YXt5fU1byPR8dN189nuQ3utgxkjNqjTR8Dbhti/yWZuWfzMWRDBdofszI7IrZYsxER\nuwGz23xvJUycNIGHFvU9t71o8RImTpxQsKL6227yS9lx9524a/6dpUupn4Ttv/0VJl1yJpsfdlDp\namrN+1h6voiYDLwdGLYR0q52u4G+TKPB8nZgGnAecPS6Do6IGcAMgOjdkp6eTV9snaqZcZuM45Nn\n/TPfO/k7rHxyZelyaqfvvcfT/8gyerbeiu1nfoVV9z3EMzf9oXRZteN9rG4xmrOBWn/HN83MzJkt\n26cBnwI2H+I074yIvwLuBI7PzCH7UNtqrGTm5RExFriy+eF/l5nr/DOiWfRMgDEbTarEsJ++xUuZ\nMnnic9uTJ21PX9/SghXVV++YXj5x1qf55aXX8NufXV+6nFrqf2QZAAOPPcHTV/2ajV81zcbKCPM+\n1oaq9Xf8YBExHXgkM2+KiP3XcYrZwMWZ+eeI+BBwLvDmoT5zyG6giPhGRJweEacDfwNsCdwHfLT5\nWm3cOG8+O++8I1OnTmHs2LEcfvihzJ5zZemyaunD//qPLL77IeZ857LSpdRSjB9HbDL+uefj99uL\nZ+++v2xRNeR9LK3VfwMOiYj7ge8Db46IC1oPyMxlmfnn5uZ3gNcOd9LhkpV5zf/fFvhjyzYMHe9U\nTn9/Px877kTmXn4RvT09nHPuJSxcaB/0SPsve+/KX7/zTTxw2/2cOvffAbjo1Av43dU3Fa6sPnq3\n2YqXndaYPRG9vTw592pWXjdvmHdpfXgfj45Tz/oi++y3F1ttvRVX/W42Z546kx9fVKvhkiOmW5bb\nz8zPAJ8BaCYrn8zM97QeExHbZ+aS5uYhDD0Qt/GebGNydkTcDLw3M//Q3D4KOC4zXzfce6vSDVRl\nf7f93qVLqL2vbuF4hNHwzyvGly6h9hY+Y/f2aFjw8A2j2nq4YeLfj9rv2tf1/bitn62lsTI9Ik4G\n5mXmZRHxFRqNlNXAY8CHM/P2oc7V7gDbw4AfRsS7gTcC/wAc0OZ7JUlSB3VjKpCZvwB+0Xx+Usvr\nz6Uv7Wp3gO29EXEkcCnwIHBAZvqnpiRJ6rghGysR8Qee32DbGugFbogIMvO/drI4SZI0vG4Zs9Ip\nwyUr04fZL0mS1FFDNlYy84HRKkSSJL0wo7koXAntLrcvSZJUhN+6LElSxQ2ULqDDTFYkSVJXM1mR\nJKniEsesSJIkFWOyIklSxQ104xK2I8hkRZIkdTWTFUmSKm7AMSuSJEnl2FiRJEldzW4gSZIqzqnL\nkiRJBZmsSJJUcS63L0mSVJDJiiRJFeeYFUmSpIJMViRJqjjHrEiSJBVksiJJUsWZrEiSJBVksiJJ\nUsU5G0iSJKkgkxVJkipuoN7BismKJEnqbiYrkiRV3IBjViRJksqxsSJJkrqa3UCSJFVcli6gw0xW\nJElSVzNZkSSp4lxuX5IkqSCTFUmSKm4gnLosSZJUjMmKJEkV52wgSZKkgkxWJEmqOGcDSZIkFWSy\nIklSxQ3UezKQyYokSepuJiuSJFXcAPWOVkxWJElSVzNZkSSp4lxnRZIkqSAbK5IkqavZDVQDP1ky\nr3QJtfeTJaUr2DAsO3rX0iXU3jYXLipdgjrAqcuSJEkFmaxIklRxLrcvSZJUkMmKJEkV59RlSZKk\ngkxWJEmqOGcDSZIkFWSyIklSxTkbSJIkqSCTFUmSKs5kRZIkqSCTFUmSKi6dDSRJklSOyYokSRXn\nmBVJkqSCbKxIkqSuZjeQJEkVZzeQJElSQSYrkiRVXJYuoMNMViRJUlczWZEkqeIGXBROkiSpHJMV\nSZIqztlAkiRJBZmsSJJUcSYrkiRJBZmsSJJUca6zIkmSVJDJiiRJFec6K5IkSQWZrEiSVHHOBpIk\nSSrIxookSepqNlYkSaq4HMXHUCJiXET8NiJuiYgFEfEvazlm44i4JCLujogbImLqcD+fjRVJkjRS\n/gy8OTNfDewJvDUiXj/omA8Cj2fmzsC/A18d7qQOsJUkqeIGumRZuMxM4Mnm5tjmY3BxhwJfaD7/\nIXBGRETzvWtlsiJJktoWETMiYl7LY8ag/b0RMR94BPiPzLxh0CkmAQ8BZOZqYDmwzVCfabIiSVLF\njebU5cycCcwcYn8/sGdEbAX8JCJelZm3vpjPNFmRJEkjLjOfAK4G3jpo12JgCkBEjAG2BJYNdS4b\nK5IkVVwXzQbarpmoEBHjgbcAtw867DLgvc3nhwH/d6jxKmA3kCRJGjnbA+dGRC+NQGRWZs6JiJOB\neZl5GXA2cH5E3A08Bhw53EltrEiSVHHdstx+Zv4eeM1aXj+p5fkzwLvW57x2A0mSpK5msiJJUsUN\nROkKOstkRZIkdTWTFUmSKq5bVrDtFJMVSZLU1UxWJEmquHrnKiYrkiSpy9lYkSRJXc3GSosDD9if\nBbdey+0Lf8WnTji2dDm15XXuPK/xKBi/KZt85CQ2+/J32eyUs+l9xa6lK6od7+P2DYziowQbK009\nPT2c/vVTmH7we9jj1W/iiCPewa677lK6rNrxOnee13h0jD/6WFbdeiNPfvYDPHnSh+jve7B0SbXi\nfaxWNlaa9t3nNdxzz/3cd9+DrFq1ilmzfsohBx9Yuqza8Tp3ntd4FIzflDGv3INV117R2O5fDSuf\nKltTzXgfr58BctQeJbQ1GygivsdaBhtn5gdGvKJCJk6awEOL+p7bXrR4Cfvu8xdfb6AXyevceV7j\nzuvZdgIDf1rO+A+eQO+UV9D/wJ2svPCb8OwzpUurDe9jtWo3WZkDXN58XAVsATy5roMjYkZEzIuI\neQMD/rUhqV6it5feHXbh2atn8+QXjiH//Awbv33YL46VOiZH8VFCW8lKZv6odTsiLgZ+NcTxM4GZ\nAGM2mlSJ6d99i5cyZfLE57YnT9qevr6lBSuqJ69z53mNO2/gsUfJxx+l/97bAVh147Vs/PajCldV\nL97HavVCx6zsArx0JAsp7cZ589l55x2ZOnUKY8eO5fDDD2X2nCtLl1U7XufO8xp3Xq54nIHHHqVn\nwmQAxuy2FwN9DxSuql68j9dP3WcDDZusREQA/Ty/22cp8M+dKqqE/v5+Pnbcicy9/CJ6e3o459xL\nWLjwztJl1Y7XufO8xqNj5QVnMH7GZ4gxYxl4dAlPn31q6ZJqxftYrSJz+F6aiLg1M1/1Qj6gKt1A\nkspbdrRrlXTaNhfeVrqEDcLqZxfHaH7ex6ceOWq/a792//dH9WeD9ruBboqIfTpaiSRJ0lq0+0WG\nrwOOjogHgKeAADIz/2vHKpMkSW2pexdGu40VV+KRJElFtDt12WHukiR1qVKzdEaLy+1LkqSu1m43\nkCRJ6lJZ81ErJiuSJKmr2VjysBwOAAALVUlEQVSRJEldzW4gSZIqzgG2kiRJBZmsSJJUcQMOsJUk\nSSrHZEWSpIqrd65isiJJkrqcyYokSRXnmBVJkqSCTFYkSao411mRJEkqyGRFkqSK84sMJUmSCjJZ\nkSSp4hyzIkmSVJDJiiRJFeeYFUmSpIJsrEiSpK5mN5AkSRXnAFtJkqSCTFYkSaq4gXSArSRJUjEm\nK5IkVVy9cxWTFUmS1OVMViRJqriBmmcrJiuSJKmrmaxIklRxLrcvSZJUkMmKJEkV5wq2kiRJBZms\nSJJUcc4GkiRJKshkRZKkinM2kCRJUkE2ViRJUlezG0iSpIpz6rIkSVJBJiuSJFVcpgNsJUmSijFZ\nkSSp4lwUTpIkqSCTFUmSKq7us4FsrEhtmPaSyaVL2CBscurM0iXU3rS5R5UuQVpvNlYkSao4l9uX\nJEkqyGRFkqSKczaQJElSQSYrkiRVnCvYSpIkFWSyIklSxdV9nRWTFUmS1NVMViRJqjjXWZEkSSrI\nxookSepqdgNJklRxLgonSZJUkMmKJEkV56JwkiRJbYqI70bEIxFx6zr27x8RyyNifvNx0nDnNFmR\nJKniumzMyjnAGcB5Qxzzy8yc3u4JTVYkSdKIycxrgcdG8pw2ViRJqrgcxf9FxIyImNfymPECSn5D\nRNwSEVdExO7DHWw3kCRJaltmzgRmvohT3AzskJlPRsRBwKXALkO9wWRFkqSKG8gctceLlZkrMvPJ\n5vO5wNiI2Hao99hYkSRJoyYiJkRENJ/vS6Mtsmyo99gNJElSxXXTXKCIuBjYH9g2IhYBnwfGAmTm\nWcBhwIcjYjWwEjgyh1koxsaKJEkaMZl51DD7z6AxtbltNlYkSaq4LltnZcQ5ZkWSJHU1kxVJkirO\nZEWSJKkgGyuSJKmr2Q0kSVLFDTPzt/JMViRJUlczWZEkqeIcYCtJklSQyYokSRWXJiuSJEnlmKxI\nklRxzgaSJEkqyGRFkqSKczaQJElSQSYrkiRVnGNWJEmSCjJZkSSp4hyzIkmSVJDJiiRJFecKtpIk\nSQXZWJEkSV3NbiBJkipuwKnLkiRJ5ZisSJJUcQ6wlSRJKsjGSosDD9ifBbdey+0Lf8WnTji2dDm1\n5XXurC+ediLXLriCS6+5qHQptXb+rEt5x3uO4dCjP8T5l/ykdDm15L3cvoHMUXuUYGOlqaenh9O/\nfgrTD34Pe7z6TRxxxDvYddddSpdVO17nzrv0+3P40JHHlS6j1u66935+dNnPuPg7p/Gjc7/JNb/+\nLQ8u6itdVu14L2sNGytN++7zGu65537uu+9BVq1axaxZP+WQgw8sXVbteJ0776br57P8iRWly6i1\ne+9/iD12n8b4ceMYM6aXvffcg/+85rrSZdWO93L7chT/V4KNlaaJkybwUMtfRosWL2HixAkFK6on\nr7PqYOedduDmWxbwxPIVrHzmGX75mxtZ+vCjpcuSaqut2UARcXJmntSy3Qucl5lHr+P4GcAMgOjd\nkp6eTUeiVknqCq+Y+nI+cPS7mHH85xg/bhzTdtmJnh7/9lM5rrPSMCUiPgMQERsDPwbuWtfBmTkz\nM/fOzL2r0lDpW7yUKZMnPrc9edL29PUtLVhRPXmdVRfvPPhAZn33G5z7zVPZYvPNmfryyaVLkmqr\n3cbKB4A9mg2W2cDVmfmFjlVVwI3z5rPzzjsydeoUxo4dy+GHH8rsOVeWLqt2vM6qi2WPPwHAkqWP\ncNU113HQW/YvW5A2aHUfszJkN1BE7NWy+XXg28B1wLURsVdm3tzJ4kZTf38/HzvuROZefhG9PT2c\nc+4lLFx4Z+myasfr3HmnnvVF9tlvL7baeiuu+t1szjx1Jj++aHbpsmrn+M9+iSdWrGDMmDF87hMf\nYYvNNytdUu14L2uNyCH6uSLi6ubTNQdFy3My883DfcCYjSbVuyNNG4RpLzHiHw3zF1xcuoTa23P3\no0qXsEFY8PANMZqf94pt9xq137X3/PHmUf3ZYJhkJTPfBBARn1/b7o5UJEmS1KLd7wZ6suX5OGA6\ncNvIlyNJktZX3b8bqK3GSmb+W+t2RPxv4OcdqUiSJKnFC10YYBPATnxJktRx7S4K9wf+/xiVXmA7\n4OROFSVJktqXOVC6hI5qd8zK9Jbnq4GHM3N1B+qRJEl6nnbHrDzQ6UIkSdILM1DzAbZ+mYUkSepq\n7XYDSZKkLjXUAq91YLIiSZK6msmKJEkV55gVSZKkgkxWJEmqOMesSJIkFWSyIklSxQ2YrEiSJJVj\nsiJJUsWls4EkSZLKMVmRJKninA0kSZJUkI0VSZLU1ewGkiSp4lxuX5IkqSCTFUmSKs4BtpIkSQWZ\nrEiSVHEuty9JklSQyYokSRXnmBVJkqSCTFYkSao411mRJEkqyGRFkqSKc8yKJElSQSYrkiRVnOus\nSJIkFWSyIklSxaWzgSRJksqxsSJJkrqa3UCSJFWcA2wlSZIKMlmRJKniXBROkiSpIJMVSZIqzqnL\nkiRJBZmsSJJUcY5ZkSRJKshkRZKkijNZkSRJalNEvDUi7oiIuyPi02vZv3FEXNLcf0NETB3unDZW\nJEmquBzFx1Aiohc4E3gbsBtwVETsNuiwDwKPZ+bOwL8DXx3u57OxIkmSRsq+wN2ZeW9mPgt8Hzh0\n0DGHAuc2n/8Q+JuIiKFO2vExK6ufXTxkAd0oImZk5szSddSZ17jzvMajo2rXecHDN5QuYb1V7RqX\nMJq/ayNiBjCj5aWZLf99JgEPtexbBLxu0CmeOyYzV0fEcmAb4I/r+kyTlbWbMfwhepG8xp3nNR4d\nXufO8xp3kcycmZl7tzw63pC0sSJJkkbKYmBKy/bk5mtrPSYixgBbAsuGOqmNFUmSNFJuBHaJiB0j\nYiPgSOCyQcdcBry3+fww4P/mMHOvXWdl7ewb7Tyvced5jUeH17nzvMYV0RyD8lHg50Av8N3MXBAR\nJwPzMvMy4Gzg/Ii4G3iMRoNmSFH3hWQkSVK12Q0kSZK6mo0VSZLU1Tb4xkpETI2IW0vXsaGIiH+K\niNsi4sLStdTN2u7liHhfRJxRqiZppETELyJi79J1qAwH2Gq0fQT428xctOaFiBiTmasL1iStt+aK\nm5GZA6Vrkepug09WmsZExIXNv/h/GBGbRMRrI+KaiLgpIn4eEduXLrLqIuIsYCfgiohYHhHnR8R1\nwPmFS6udiNgpIn4HbNfy2tsj4jcRsW3B0iqtmV7dERHnAbcC/S37DouIc4oVVxODE8KI+GREfKFl\nuycizomILxUpUEXYWGmYBnwzM3cFVgDHAt8ADsvM1wLfBU4pWF8tZOYxQB/wJhpfXrUbjZTlqKKF\n1UxETAN+BLwPeLT52t8BnwYOysx1LmmttuxC49+L3YGnShezgRkDXAjclZknli5Go8duoIaHMvO6\n5vMLgM8CrwL+o/ndSr3AkkK11dllmbmydBE1sx3wU+DvM3NhRLwGeDOwN3BAZq4oWl09PJCZ15cu\nYgP1bWBWZvrH4wbGZKVh8GIzfwIWZOaezccemXlAicJqzr9KR95y4EHgv7e8dg+wOfDKIhXVT+t9\n2/pvx7jRLqSmVvP8302t1/XXwJsiwmu9gbGx0vDyiHhD8/m7geuB7da8FhFjI2L3YtVJ7XsW+Dvg\nHyLi3c3XHgDeCZznfTziHo6IXSOih8Z114v3MPDSiNgmIjYGprfsOxuYC8xqfqeMNhA2VhruAI6N\niNuAl9AcrwJ8NSJuAeYD+xWsT2pbZj5F4x/444Etmq/dDhwN/CAiXlGwvLr5NDCHxl/8dhWPgMxc\nBZwM/Bb4D+D2Qfu/BvyOxnLt/g7bQLjcviRJ6mq2SiVJUlezsSJJkrqajRVJktTVbKxIkqSuZmNF\nkiR1NRsrkiSpq9lYkSRJXe3/AXJXzB48obfUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Проверим на случайных отрывках из тестовых текстов\n",
    "import random\n",
    "\n",
    "small_texts = []\n",
    "for text in X_test:\n",
    "    begin = random.randint(0, len(text) - 10)\n",
    "    small_texts.append(text[begin:begin+10])\n",
    "y_predicted_small = clf.predict(small_texts)\n",
    "test_classify(y_test, y_predicted_small, clf.classes_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SMuJP3Tp3Ip3"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "2_Language_Detection.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}