10_nlp_basics_lecture.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing\n",
    "\n",
    "## What is NLP\n",
    "\n",
    "> \"*Natural language processing (NLP) is a collective term referring to automatic computational processing of human languages. This includes both algorithms that take human-produced text as input, and algorithms that produce natural looking text as outputs.*\" *(Goldberg, 2017)\n",
    "\n",
    "\n",
    "## Where NLP is used?\n",
    "\n",
    "### **Text Classification**  \n",
    "  *is the process of assigning tags or categories to text according to its content.*\n",
    "\n",
    "* **Sentiment analysis**  \n",
    "    *is the process of analyzing emotions within a text and classifying them as positive, negative, or neutral*  \n",
    "    \n",
    "      Example: By running sentiment analysis on social media posts, product reviews, NPS surveys, and customer feedback, businesses can gain valuable insights about how customers perceive their brand. See this Zoom customer and product reviews, e.g.:\n",
    "     <img src=\"./pics/nlp/sentiment1.png\" style=\"width:500px;\">\n",
    "     <img src=\"./pics/nlp/sentiment2.png\" style=\"width:500px;\">\n",
    "      \n",
    "      A sentiment classifier tries to detect the emotions people express with their words, and classifies texts into Positive, Negative, or Neutral. From image above it can understand the nuance of each opinion and automatically tag 1st review as Negative and 2nd one as Positive. \n",
    "            \n",
    "            \n",
    "* **Topic classification**  \n",
    "    *is the process of identifying the main themes or topics within a text and assigning predefined tags.*  \n",
    "    For training a topic classifier, it's needed to be familiar with the data which is analyzing, so to define relevant categories.  \n",
    "      Example: you might work for a software company, and receive a lot of customer support tickets that mention technical issues, usability, and feature requests.In this case, you might define your tags as Bugs, Feature Requests, and UX/IX.\n",
    "\n",
    "* **Intent detection**  \n",
    "    *is the process of identifying the purpose, goal, or intention behind a text.*  \n",
    "      \n",
    "      Example: sorting outbound sales email responses by Interested, Need Information, Unsubscribe, Bounce.The tag Interested could help to spot a potential sale opportunity as soon as an email enters an inbox!  \n",
    "         \n",
    "         \n",
    "### Text Extraction\n",
    "  *is the process of extraction of specific information that is already in the text.*\n",
    "  \n",
    "* **Keyword extraction**  \n",
    "    *is the process of automatically extracting the most important words and expressions within a text.*  \n",
    "    This can provide with a sort of preview of the content and its main topics, without needing to read each piece. \n",
    "    \n",
    "      Example: check out this feature request, below, processed with MonkeyLearn’s public keyword extractor\n",
    "     <img src=\"./pics/nlp/extract.png\" style=\"width:500px;\">\n",
    "\n",
    "* **Named Entity Recognition (NER)**  \n",
    "    *is the process of extracting the entities such as names of people, companies, places, etc.*  \n",
    "    \n",
    "* **Machine Translation**  \n",
    "    *is the process of translating speech and text to different languages*  \n",
    "    One of the first problems addressed by NLP researchers. Online translation tools (like *Google Translate*) use different NLP techniques to achieve human-levels of accuracy in translating speech and text to different languages. Custom translators models can be trained for a specific domain to maximize the accuracy of the results. \n",
    "\n",
    "* **Topic Modeling**  \n",
    "    *is the process of finding relevant topics in a text by grouping texts with similar words and expressions.*   \n",
    "    This is similar to topic classification. \n",
    "\n",
    "* **Natural Language Generation (NLG)**  \n",
    "    *is the process of analyzing unstructured data and using it as an input to automatically create content*   \n",
    "    Can be used to generate automated answers, write emails, and even books!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tasks & Techniques\n",
    "\n",
    "**Syntactic analysis**  \n",
    "or parsing/syntax analysis, identifies the syntactic structure of a text and the dependency relationships between words, represented on a diagram called a parse tree.\n",
    "\n",
    "**Semantic analysis**  \n",
    "focuses on identifying the meaning of language. However, since language is polysemic and ambiguous, semantics is considered one of the most challenging areas in NLP. Semantic tasks analyze the structure of sentences, word interactions, and related concepts, in an attempt to discover the meaning of words, as well as understand the topic of a text. \n",
    "\n",
    "* Tokenization\n",
    "* Part-of-speech tagging\n",
    "* Dependency Parsing\n",
    "* Constituency Parsing\n",
    "* Lemmatization & Stemming\n",
    "* Stopword Removal\n",
    "* Word Sense Disambiguation\n",
    "* Named Entity Recognition (NER)\n",
    "* Relationship extraction\n",
    "\n",
    "### Tokenization  \n",
    "*is an essential task in NLP used to break up a string of words into semantically useful units called **tokens**.*\n",
    "\n",
    "Sentence tokenization splits sentences within a text, and word tokenization splits words within a sentence. Generally, word tokens are separated by blank spaces, and sentence tokens by stops. However, you can perform high-level tokenization for more complex structures, like words that often go together, otherwise known as collocations (e.g., *New York*).\n",
    "\n",
    "    Example of how word tokenization simplifies text:  \n",
    "    Customer service couldn’t be better! = “customer service” “could” “not” “be” “better”. \n",
    "    \n",
    "    \n",
    "### Part-of-speech (PoS) tagging\n",
    "\n",
    "*involves adding a part of speech category to each token within a text.*  \n",
    "\n",
    "Some common PoS tags are verb, adjective, noun, pronoun, conjunction, preposition, intersection, among others. PoS tagging is useful for identifying relationships between words and, therefore, understand the meaning of sentences.\n",
    "\n",
    "    Example above would look like this:  \n",
    "    “Customer service”: NOUN, “could”: VERB, “not”: ADVERB, be”: VERB, “better”: ADJECTIVE, “!”: PUNCTUATION\n",
    "\n",
    "\n",
    "### Dependency Parsing  \n",
    "\n",
    "*refers to the way the words in a sentence are connected.* \n",
    "\n",
    "    Example: a dependency parser, therefore, analyzes how ‘head words’ are related and modified by other words too understand the syntactic structure of a sentence\n",
    "   <img src=\"./pics/nlp/dependency-parsing.png\" style=\"width:900px;\">\n",
    "\n",
    "\n",
    "### Constituency Parsing  \n",
    "\n",
    "*aims to visualize the entire syntactic structure of a sentence by identifying phrase structure grammar.*  \n",
    "\n",
    "It consists of using abstract terminal and non-terminal nodes associated to words.\n",
    "\n",
    "    Example:\n",
    "   <img src=\"./pics/nlp/constituency-parsing.png\" style=\"width:600px;\">\n",
    "\n",
    "\n",
    "### Lemmatization & Stemming  \n",
    "\n",
    "*aims to transform the words back to their root form.*\n",
    "\n",
    "The word as it appears in the dictionary – its **root form** – is called a **lemma**. When we speak or write, we tend to use inflected forms of a word (words in their different grammatical forms). To make these words easier for computers to understand, NLP uses lemmatization and stemming to transform them back to their root form.\n",
    " \n",
    "    Example: \n",
    "    The terms \"is, are, am, were, and been,” are grouped under the lemma ‘be.’ So, if we apply this lemmatization to “African elephants have four nails on their front feet,” the result will look something like this:\n",
    "\n",
    "    African elephants have four nails on their front feet = “African,” “elephant,” “have,” “4”, “nail,” “on,” “their,” “foot”]\n",
    "\n",
    "This example is useful to see how the lemmatization changes the sentence using its base form (e.g., the word \"feet\"\" was changed to \"foot\").\n",
    "\n",
    "When we refer to stemming, the root form of a word is called a stem. Stemming \"trims\" words, so word stems may not always be semantically correct.\n",
    "\n",
    "For example, stemming the words “consult,” “consultant,” “consulting,” and “consultants” would result in the root form “consult.”\n",
    "\n",
    "While lemmatization is dictionary-based and chooses the appropriate lemma based on context, stemming operates on single words without considering the context. \n",
    "\n",
    "    Example: “This is better”  \n",
    "    The word “better” is transformed into the word “good” by a lemmatizer but is unchanged by stemming. Eventhough stemmers can lead to less-accurate results, they are easier to build and perform faster than lemmatizers. But lemmatizers are recommended if seeking more precise linguistic rules.\n",
    "\n",
    "\n",
    "### Stopword Removal  \n",
    "\n",
    "*involves filtering out high-frequency words that add little or no semantic value to a sentence, for example, which, to, at, for, is, etc.*  \n",
    "\n",
    "Removing stop words is an essential step in NLP text processing. Lists of stopwords can be customized in order to include words that you want to ignore.\n",
    "\n",
    "    Example:  \n",
    "    Let’s say you want to classify customer service tickets based on their topics. \n",
    "    In this example: “Hello, I’m having trouble logging in with my new password”, it may be useful to remove stop words like “hello”, “I”, “am”, “with”, “my”, so you’re left with the words that help you understand the topic of the ticket: “trouble”, “logging in”, “new”, “password”.\n",
    "    \n",
    "\n",
    "### Word Sense Disambiguation (WSD)\n",
    "\n",
    "Depending on their context, words can have different meanings. \n",
    "\n",
    "    Example: the word “book”\n",
    "      * You should read this book; it’s a great novel!\n",
    "      * You should book the flights as soon as possible.\n",
    "      * You should close the books by the end of the year.\n",
    "      * You should do everything by the book to avoid potential complications.\n",
    "\n",
    "There are two main techniques that can be used for WSD: \n",
    "* knowledge-based (or dictionary approach)  \n",
    "tries to infer meaning by observing the dictionary definitions of ambiguous terms within a text\n",
    "* supervised approach  \n",
    "is based on NLP algorithms that learn from training data.\n",
    "\n",
    "\n",
    "### Named Entity Recognition (NER)  \n",
    "*involves extracting entities from within a text (e.g., names, places)*  \n",
    "\n",
    "Entities can be names, places, organizations, email addresses, and more.\n",
    "<img src=\"./pics/nlp/ner.png\" style=\"width:700px;\">\n",
    "\n",
    "#### Open-Source named entity recognition APIs\n",
    "\n",
    "Open-source APIs are for developers: they are free, flexible, and entail a gentle learning curve. Here are a few options:\n",
    "\n",
    "   * **Stanford Named Entity Recognizer (SNER)**  \n",
    "       JAVA tool developed by Stanford University is considered the standard library for entity extraction. It’s based on Conditional Random Fields (CRF) and provides pre-trained models for extracting person, organization, location, and other entities. \n",
    "   * **SpaCy**  \n",
    "       Python framework known for being fast and very easy to use. It has an excellent statistical system that you can use to build customized NER extractors.\n",
    "   * **Natural Language Toolkit (NLTK)**  \n",
    "       Python lib which is widely used for NLP tasks. NLKT has its own classifier to recognize named entities called ne_chunk, but also provides a wrapper to use the Stanford NER tagger in Python.\n",
    "\n",
    "\n",
    "### Relationship extraction\n",
    "\n",
    "*finds relationships between two nouns.*  \n",
    "\n",
    "    Example:  \n",
    "    In the phrase “Susan lives in Los Angeles,” a person (Susan) is related to a place (Los Angeles) by the semantic category “lives in.”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP in everyday life\n",
    "\n",
    "11 of the most common and most powerful uses of natural language processing in everyday life:\n",
    "\n",
    "* Email filters\n",
    "* Virtual assistants, voice assistants, or smart speakers  \n",
    "  * Amazon's Alexa, Yandex's Alice\n",
    "* Online search engines   \n",
    "  * Google, Yandex, ...\n",
    "* Predictive text and autocorrect\n",
    "* Monitor brand sentiment on social media\n",
    "* Sorting customer feedback\n",
    "* Automating processes in customer support\n",
    "* Chatbots\n",
    "* Automatic summarization\n",
    "* Machine translation\n",
    "* Natural language generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./pics/nlp/nltk_main.jpeg\" style=\"width:200px;\"><img src=\"./pics/nlp/spacy_main.png\" style=\"width:200px;\"><img src=\"./pics/nlp/text_main.png\" style=\"width:200px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [**SpaCy**](https://spacy.io/usage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./pics/nlp/spacy_installation.png\" style=\"width:500px;\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: spacy in /home/muha/anaconda3/lib/python3.6/site-packages (2.3.2)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (3.0.4)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (0.8.0)\n",
      "Requirement already satisfied: setuptools in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (40.0.0)\n",
      "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (1.1.3)\n",
      "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (1.0.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (4.51.0)\n",
      "Requirement already satisfied: thinc==7.4.1 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (7.4.1)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (2.0.4)\n",
      "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (1.0.4)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (1.15.0)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (1.0.4)\n",
      "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (0.4.1)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy) (2.19.1)\n",
      "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /home/muha/anaconda3/lib/python3.6/site-packages (from catalogue<1.1.0,>=0.0.7->spacy) (2.0.0)\n",
      "Requirement already satisfied: urllib3<1.24,>=1.21.1 in /home/muha/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy) (1.22)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/muha/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.0.4)\n",
      "Requirement already satisfied: idna<2.8,>=2.5 in /home/muha/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/muha/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2018.8.24)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/muha/anaconda3/lib/python3.6/site-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy) (3.4.0)\n",
      "\u001b[31mnilmtk 0.4.0.dev1-git. has requirement matplotlib==3.1.3, but you'll have matplotlib 2.2.2 which is incompatible.\u001b[0m\n",
      "\u001b[31mnilmtk 0.4.0.dev1-git. has requirement pandas==0.25.3, but you'll have pandas 0.23.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mnilmtk 0.4.0.dev1-git. has requirement scikit-learn>=0.21.2, but you'll have scikit-learn 0.19.2 which is incompatible.\u001b[0m\n",
      "\u001b[33mYou are using pip version 10.0.1, however version 20.3b1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Requirement already satisfied: en_core_web_sm==2.3.1 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.1/en_core_web_sm-2.3.1.tar.gz#egg=en_core_web_sm==2.3.1 in /home/muha/anaconda3/lib/python3.6/site-packages (2.3.1)\n",
      "Requirement already satisfied: spacy<2.4.0,>=2.3.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from en_core_web_sm==2.3.1) (2.3.2)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.4)\n",
      "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.4)\n",
      "Requirement already satisfied: thinc==7.4.1 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (7.4.1)\n",
      "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.1.3)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.19.1)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (4.51.0)\n",
      "Requirement already satisfied: setuptools in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (40.0.0)\n",
      "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.0)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.0.4)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (0.8.0)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (3.0.4)\n",
      "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (0.4.1)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /home/muha/anaconda3/lib/python3.6/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.15.0)\n",
      "Requirement already satisfied: idna<2.8,>=2.5 in /home/muha/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/muha/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2018.8.24)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/muha/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (3.0.4)\n",
      "Requirement already satisfied: urllib3<1.24,>=1.21.1 in /home/muha/anaconda3/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.22)\n",
      "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /home/muha/anaconda3/lib/python3.6/site-packages (from catalogue<1.1.0,>=0.0.7->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.0.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/muha/anaconda3/lib/python3.6/site-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (3.4.0)\n",
      "\u001b[31mnilmtk 0.4.0.dev1-git. has requirement matplotlib==3.1.3, but you'll have matplotlib 2.2.2 which is incompatible.\u001b[0m\n",
      "\u001b[31mnilmtk 0.4.0.dev1-git. has requirement pandas==0.25.3, but you'll have pandas 0.23.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mnilmtk 0.4.0.dev1-git. has requirement scikit-learn>=0.21.2, but you'll have scikit-learn 0.19.2 which is incompatible.\u001b[0m\n",
      "\u001b[33mYou are using pip version 10.0.1, however version 20.3b1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the model via spacy.load('en_core_web_sm')\n",
      "\u001b[2K\u001b[38;5;2m✔ Loaded compatibility table\u001b[0m\n",
      "\u001b[1m\n",
      "====================== Installed models (spaCy v2.3.2) ======================\u001b[0m\n",
      "\u001b[38;5;4mℹ spaCy installation:\n",
      "/home/muha/anaconda3/lib/python3.6/site-packages/spacy\u001b[0m\n",
      "\n",
      "TYPE      NAME             MODEL            VERSION                            \n",
      "package   en-core-web-sm   en_core_web_sm   \u001b[38;5;2m2.3.1\u001b[0m   \u001b[38;5;2m✔\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#!pip install -U spacy\n",
    "#!python -m spacy download en_core_web_sm\n",
    "#!python -m spacy validate\n",
    "\n",
    "import sys\n",
    "!{sys.executable} -m pip install spacy\n",
    "!{sys.executable} -m spacy download en_core_web_sm\n",
    "!{sys.executable} -m spacy validate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[SpaCy linguistic features](https://spacy.io/usage/linguistic-features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple 0 5 ORG\n",
      "U.K. 27 31 GPE\n",
      "$1 billion 44 54 MONEY\n"
     ]
    }
   ],
   "source": [
    "# Named Entity Recognition\n",
    "doc = nlp(\"Apple is looking at buying U.K. startup for $1 billion\")\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.start_char, ent.end_char, ent.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'U.K.'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"Apple is looking at buying U.K. startup for $1 billion\"[27:31]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./pics/nlp/spacy_ner.png\" style=\"width:900px;\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http:https://www.w3.org/2000/svg\" xmlns:xlink=\"http:https://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"e9c82ba4b01140298e3d084d7bacdcfb-0\" class=\"displacy\" width=\"1975\" height=\"399.5\" direction=\"ltr\" style=\"max-width: none; height: 399.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Apple</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">is</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">looking</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">at</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">buying</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"925\">U.K.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"925\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">startup</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1275\">for</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1275\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1450\">$</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1450\">SYM</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1625\">1</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1625\">NUM</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1800\">billion</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1800\">NUM</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-0\" stroke-width=\"2px\" d=\"M70,264.5 C70,89.5 395.0,89.5 395.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,266.5 L62,254.5 78,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-1\" stroke-width=\"2px\" d=\"M245,264.5 C245,177.0 390.0,177.0 390.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M245,266.5 L237,254.5 253,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-2\" stroke-width=\"2px\" d=\"M420,264.5 C420,177.0 565.0,177.0 565.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M565.0,266.5 L573.0,254.5 557.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-3\" stroke-width=\"2px\" d=\"M595,264.5 C595,177.0 740.0,177.0 740.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">pcomp</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M740.0,266.5 L748.0,254.5 732.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-4\" stroke-width=\"2px\" d=\"M945,264.5 C945,177.0 1090.0,177.0 1090.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M945,266.5 L937,254.5 953,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-5\" stroke-width=\"2px\" d=\"M770,264.5 C770,89.5 1095.0,89.5 1095.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-5\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1095.0,266.5 L1103.0,254.5 1087.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-6\" stroke-width=\"2px\" d=\"M770,264.5 C770,2.0 1275.0,2.0 1275.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-6\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1275.0,266.5 L1283.0,254.5 1267.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-7\" stroke-width=\"2px\" d=\"M1470,264.5 C1470,89.5 1795.0,89.5 1795.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-7\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">quantmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1470,266.5 L1462,254.5 1478,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-8\" stroke-width=\"2px\" d=\"M1645,264.5 C1645,177.0 1790.0,177.0 1790.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-8\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1645,266.5 L1637,254.5 1653,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-9\" stroke-width=\"2px\" d=\"M1295,264.5 C1295,2.0 1800.0,2.0 1800.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e9c82ba4b01140298e3d084d7bacdcfb-0-9\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1800.0,266.5 L1808.0,254.5 1792.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Spacy Visualizer\n",
    "from spacy import displacy\n",
    "doc = nlp(\"Apple is looking at buying U.K. startup for $1 billion\")\n",
    "displacy.render(doc, style=\"dep\", jupyter=True)\n",
    "#displacy.serve(doc, style=\"dep\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Apple\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " is looking at buying \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    U.K.\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " startup for \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    $1 billion\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
       "</mark>\n",
       "</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc = nlp(\"Apple is looking at buying U.K. startup for $1 billion\")\n",
    "displacy.render(doc, style=\"ent\", jupyter=True)\n",
    "#displacy.serve(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple\n",
      "is\n",
      "looking\n",
      "at\n",
      "buying\n",
      "U.K.\n",
      "startup\n",
      "for\n",
      "$\n",
      "1\n",
      "billion\n"
     ]
    }
   ],
   "source": [
    "# Tokenization\n",
    "doc = nlp(\"Apple is looking at buying U.K. startup for $1 billion\")\n",
    "for token in doc:\n",
    "    print(token.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [NLTK](https://www.nltk.org/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/muha/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /home/muha/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n",
      "[nltk_data] Downloading package maxent_ne_chunker to\n",
      "[nltk_data]     /home/muha/nltk_data...\n",
      "[nltk_data]   Package maxent_ne_chunker is already up-to-date!\n",
      "[nltk_data] Downloading package words to /home/muha/nltk_data...\n",
      "[nltk_data]   Package words is already up-to-date!\n",
      "[nltk_data] Downloading package treebank to /home/muha/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/treebank.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "nltk.download('maxent_ne_chunker')\n",
    "nltk.download('words')\n",
    "nltk.download('treebank')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Apple',\n",
       " 'is',\n",
       " 'looking',\n",
       " 'at',\n",
       " 'buying',\n",
       " 'U.K.',\n",
       " 'startup',\n",
       " 'for',\n",
       " '$',\n",
       " '1',\n",
       " 'billion']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"Apple is looking at buying U.K. startup for $1 billion\"\n",
    "tokens = nltk.word_tokenize(sentence)\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Apple', 'NNP'),\n",
       " ('is', 'VBZ'),\n",
       " ('looking', 'VBG'),\n",
       " ('at', 'IN'),\n",
       " ('buying', 'VBG'),\n",
       " ('U.K.', 'NNP'),\n",
       " ('startup', 'NN'),\n",
       " ('for', 'IN'),\n",
       " ('$', '$'),\n",
       " ('1', 'CD'),\n",
       " ('billion', 'CD')]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagged = nltk.pos_tag(tokens)\n",
    "tagged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "entities = nltk.chunk.ne_chunk(tagged)\n",
    "#entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display parse trees\n",
    "from nltk.corpus import treebank\n",
    "\n",
    "t = treebank.parsed_sents('wsj_0001.mrg')[0]\n",
    "t.draw()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word embeddings\n",
    "\n",
    "**Word embedding**  \n",
    "is the collective name for a set of language modeling and feature learning techniques in NLP where words or phrases from the vocabulary are mapped to vectors of real numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given this vocabulary of 10,000 words, what’s the simplest way to represent each word numerically?\n",
    "\n",
    "* One-hot vector encoding\n",
    "\n",
    "<div style=\"width:image width px; \n",
    "            font-size:80%; \n",
    "            text-align:center; \n",
    "            float: left; padding-left-right-top-bottom:0.5em;  \n",
    "            border-style: solid; border-color: rgba(211, 211, 211, 0.000);\n",
    "            background-color: rgba(0, 0, 0, 0.000);\">\n",
    "    <img src=\"./pics/nlp/emb1.jpeg\" \n",
    "         alt=\"alternate text\" \n",
    "         width=250 \n",
    "         style=\"padding-bottom:0.5em;\"/>\n",
    "    <div style=\"padding: 3px; \n",
    "                width: 250px; \n",
    "                word-wrap: break-word; \n",
    "                text-align:justify;\">\n",
    "        Our vocabulary of 10,000 words. <br> \n",
    "        <a href=\"https://towardsdatascience.com/why-do-we-use-embeddings-in-nlp-2f20e1b632d2\" \n",
    "           style=\"float: left;\"> \n",
    "           Source \n",
    "        </a>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "<div style=\"width:image width px; \n",
    "            font-size:80%; \n",
    "            text-align:center; \n",
    "            float: left; padding-left-right-top-bottom:0.5em;  \n",
    "            border-style: solid; border-color: rgba(211, 211, 211, 0.000);\n",
    "            background-color: rgba(0,0, 0, 0.000;\">\n",
    "    <img src=\"./pics/nlp/emb2.jpeg\" \n",
    "         alt=\"alternate text\" \n",
    "         width=315\n",
    "         style=\"padding-bottom:0.5em;\"/>\n",
    "    <div style=\"padding: 3px; \n",
    "                width: 315px; \n",
    "                word-wrap: break-word; \n",
    "                text-align:justify;\">\n",
    "        Our vocabulary with each word assigned an index. <br> \n",
    "        <a href=\"https://towardsdatascience.com/why-do-we-use-embeddings-in-nlp-2f20e1b632d2\" \n",
    "           style=\"float: left;\"> \n",
    "           Source \n",
    "        </a>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "<p>&nbsp;</p>\n",
    "\n",
    "Given this word-to-integer mapping, we could then represent a word as a vector of numbers as follows:      \n",
    "   * Each word will be represented as an n-dimensional vector, where n is the vocabulary size  \n",
    "   * Each word’s vector representation will be mostly “0”, except there will be a single “1” entry in the position corresponding to the word’s index in the vocabulary.  \n",
    "\n",
    "So, some examples:  \n",
    "   * The vector representation for our first vocabulary word “aardvark” will be [1, 0, 0, 0, …, 0], which is a “1” in the first position followed by 9,999 zeroes.\n",
    "   * The vector representation for our second vocabulary word “ant” will be [0, 1, 0, 0, …, 0], which is a “0” in the first position, a “1” in the second position, and 9,998 afterwards.\n",
    "   * And so on.\n",
    "\n",
    "This process is called **one-hot vector encoding**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**\n",
    "\n",
    "Now, say our NLP project is building a translation model and we want to translate the English input sentence **“the cat is black”** into another language. We first need to represent each word with a one-hot encoding. We would first look up the index of the first word, “the”, and find that its index in our 10,000-long vocabulary list is 8676.\n",
    "\n",
    "\n",
    "\n",
    "<div style=\"width:image width px; \n",
    "            font-size:80%; \n",
    "            text-align:center; \n",
    "            float: left; padding-left-right-top-bottom:0.5em;  \n",
    "            border-style: solid; border-color: rgba(211, 211, 211, 0.000);\n",
    "            background-color: rgba(0, 0, 0, 0.000);\">\n",
    "    <img src=\"./pics/nlp/emb3.jpeg\" \n",
    "         alt=\"alternate text\" \n",
    "         width=750 \n",
    "         style=\"padding-bottom:0.5em;\"/>\n",
    "    <div style=\"padding: 3px; \n",
    "                width: 750px; \n",
    "                word-wrap: break-word; \n",
    "                text-align:justify;\">\n",
    "        We first look up the index of the first word, “the”, and find that its index in our 10,000-long vocabulary list is 8676. <br> \n",
    "        <a href=\"https://towardsdatascience.com/why-do-we-use-embeddings-in-nlp-2f20e1b632d2\" \n",
    "           style=\"float: left;\"> \n",
    "           Source \n",
    "        </a>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "<div style=\"width:image width px; \n",
    "            font-size:80%; \n",
    "            text-align:center; \n",
    "            float: left; padding-left-right-top-bottom:0.5em;  \n",
    "            border-style: solid; border-color: rgba(211, 211, 211, 0.000);\n",
    "            background-color: rgba(0, 0, 0, 0.000);\">\n",
    "    <img src=\"./pics/nlp/emb4.jpeg\" \n",
    "         alt=\"alternate text\" \n",
    "         width=750 \n",
    "         style=\"padding-bottom:0.5em;\"/>\n",
    "    <div style=\"padding: 3px; \n",
    "                width: 750px; \n",
    "                word-wrap: break-word; \n",
    "                text-align:justify;\">\n",
    "        We could then represent the word “the” using a length 10,000 vector, where every entry is a 0 aside from the entry at position 8676, which is a 1. <br> \n",
    "        <a href=\"https://towardsdatascience.com/why-do-we-use-embeddings-in-nlp-2f20e1b632d2\" \n",
    "           style=\"float: left;\"> \n",
    "           Source \n",
    "        </a>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "<div style=\"width:image width px; \n",
    "            font-size:80%; \n",
    "            text-align:center; \n",
    "            float: left; padding-left-right-top-bottom:0.5em;  \n",
    "            border-style: solid; border-color: rgba(211, 211, 211, 0.000);\n",
    "            background-color: rgba(0, 0, 0, 0.000);\">\n",
    "    <img src=\"./pics/nlp/emb5.gif\" \n",
    "         alt=\"alternate text\" \n",
    "         width=750\n",
    "         style=\"padding-bottom:0.5em;\"/>\n",
    "    <div style=\"padding: 3px; \n",
    "                width: 750px; \n",
    "                word-wrap: break-word; \n",
    "                text-align:justify;\">\n",
    "        We do this index look-up for every word in the input sentence, and create a vector to represent each input word. The whole process looks a bit like this. <br> \n",
    "        <a href=\"https://towardsdatascience.com/why-do-we-use-embeddings-in-nlp-2f20e1b632d2\" \n",
    "           style=\"float: left;\"> \n",
    "           Source \n",
    "        </a>\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These one-hot vectors are a quick and easy way to represent words as vectors of real-valued numbers. \n",
    "Note that this process has generated a very sparse (mostly zero) feature vector for each input word (here, the terms “feature vector”, “embedding”, and “word representation” are used interchangeably).\n",
    "\n",
    "**The problems with sparse one-hot encodings**  \n",
    "   * **The similarity issue.**  \n",
    "       Ideally we would want similar words like “cat” and “tiger” to have somewhat similar features. But with these one-hot vectors, “cat” is as similar to “tiger” as literally any other word, which isn’t great. A related point is that we might want to do analogy-like vector operations on the word embeddings (e.g. what is “cat” - “small” + “large” equal to? Hopefully, something like a big cat, for instance “tiger” or “lion”). We’d need a sufficiently rich word representation to allow for such operations.\n",
    "   * **The vocabulary size issue.**  \n",
    "       With this approach, as you increase your vocabulary by n, your feature size vectors also increase by length n. One-hot vector dimensionality is the same as number of words. There’s reasons why you don’t want your feature size to explode —namely, more features means more parameters to estimate, and you require exponentially more data to estimate those parameters well enough to build a reasonably generalisable model (see: curse of dimensionality). As a rough rule of thumb — you want orders of magnitude more training data than you have features.\n",
    "   * **The computational issue.**  \n",
    "       Each word’s embedding/feature vector is mostly zeroes, and many machine learning models won’t work well with very high dimensional and sparse features. Neural networks in particular struggle with this type of data. With such a large feature space, you are also in danger of running into memory and even storage concerns, especially if the models you’re working with don’t play nicely with compressed versions of sparse matrices.  \n",
    "       \n",
    "       \n",
    "**Towards dense, semantically-meaningful representation**  \n",
    "Let’s now discuss what it means to represent words using dense, semantically-meaningful feature vectors.\n",
    "If we take 5 example words from our vocabulary (say… the words “aardvark”, “black”, “cat”, “duvet” and “zombie”) and examine their embedding vectors created by the one-hot encoding method discussed above, the result would look like this:\n",
    "\n",
    "<img src=\"./pics/nlp/emb6.jpeg\" style=\"width:600px;\">  \n",
    "\n",
    "*Word vectors using one-hot encoding. Each word is represented by a vector that is mostly zeroes, except there is a single “1” in the position dictated by that word’s index in the vocabulary. Note: it’s not that “black”, “cat”, and “duvet” have the same feature vector, it just looks like it here.*\n",
    "\n",
    "But, as humans speaking some language, we know that words are these rich entities with many layers of connotation and meaning. Let’s hand-craft some semantic features for these 5 words. Specifically, let’s represent each word as having some sort of value between 0 and 1 for four semantic qualities, “animal”, “fluffiness”, “dangerous”, and “spooky”:\n",
    "<img src=\"./pics/nlp/emb7.jpeg\" style=\"width:500px;\">  \n",
    "*Hand-crafted semantic features for 5 words in the vocabulary.*\n",
    "\n",
    "So, to explain a couple of examples:  \n",
    "   * Given the word “aardvark”, I’ve given it a high value for the feature “animal” (since it’s very much an animal), and relatively low values for “fluffiness” (aarvarks have short bristles), “dangerous” (they’re small, nocturnal burrowing pigs), and “spooky” (they’re charming).\n",
    "   * Given the word “cat”, I’ve given it a high value for the features “animal” and “fluffiness” (self-explanatory), a medium value for “dangerous” (self-explanatory if you’ve ever had a pet cat), and a medium value for “spooky” (try doing an image search for “sphynx cat”).\n",
    "\n",
    "**Plotting words based on semantic feature values**\n",
    "\n",
    "Each semantic feature can be though of as a single dimension in the broader, higher-dimensional semantic space.\n",
    "   * In the above made-up dataset, there are four semantic features, and we can plot two of these at a time as a 2D scatter plot (see below). Each feature is a different axis/dimension.\n",
    "   * The coordinates of each word within this space are given by its specific values on the features of interest. For example, the coordinates of the word “aardvark” on the 2D plot of fluffiness vs. animal 2D plot are (x=0.97, y=0.03).\n",
    "\n",
    "<img src=\"./pics/nlp/emb8.gif\" style=\"width:750px;\">  \n",
    "\n",
    "*Plotting word feature values on either 2 or 3 axes.*  \n",
    "\n",
    "   * Similarly, we could consider the three features (“animal”, “fluffiness” and “dangerous”) and plot the position of words in this 3D semantic space. For example, the coordinates of the word “duvet” are (x=0.01, y=0.84, z=0.12), indicating that “duvet” is highly associated with the concept of fluffiness, is maybe slightly dangerous, and not an animal.\n",
    "\n",
    "This is a hand-crafted toy example, but actual embedding algorithms will of course automatically generate embedding vectors for all the words in an input corpus. If you’d like, you can think of word embedding algorithms like word2vec as unsupervised feature extractors for words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "   [1] Goldberg, Y. (2017). Neural Network Methods for Natural Language Processing. Morgan & Claypool Publishers. [Amazon](https://www.amazon.com/Language-Processing-Synthesis-Lectures-Technologies/dp/1627052984/ref=as_li_ss_tl)  \n",
    "   [2] https://monkeylearn.com/blog/\n",
    "   \n",
    "   \n",
    "   [3] Natasha Latysheva. Why do we use word embeddings in NLP? Towards Data Science.  \n",
    "   https://towardsdatascience.com/why-do-we-use-embeddings-in-nlp-2f20e1b632d2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}