Natural Language Processing(NLP)  in Julia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Natural Language Processing in Julia\n",
    "\n",
    "- Natural Language Process == understanding(processing) everyday language.\n",
    "\n",
    "\n",
    "#### Packages\n",
    "+ Pkg.add(\"TextAnalysis\")\n",
    "+ Pkg.clone(\"WordTokenizers\")\n",
    "+ NLTK via PyCall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Natural language processing is a form of artificial intelligence that helps computers read and respond by simulating the human ability to understand everyday language. Many organizations use NLP techniques to optimize customer support,improve the efficiency of text analytics by easily finding the information they need, and enhance social media monitoring. For example, banks might implement NLP algorithms to optimize customer support; a large consumer products brand might combine natural language processing and semantic analysis to improve their knowledge management strategies and social media monitoring\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading/Creating A Document\n",
    "+ + Uses Hierarchy File->String->Tokens->Ngrams\n",
    "+ Document()\n",
    "+ FileDocument(filepath)\n",
    "+ StringDocument(\"my string\")\n",
    "+ TokenDocument(String[])\n",
    "+ NGramDocument(Dict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using TextAnalysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The best error message is the one that never shows up.\\nYou Learn More From Failure Than From Success. \\nThe purpose of software engineering is to control complexity, not to create it\""
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mystr = \"\"\"The best error message is the one that never shows up.\n",
    "You Learn More From Failure Than From Success. \n",
    "The purpose of software engineering is to control complexity, not to create it\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.StringDocument"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Basic Way\n",
    "sd1 = Document(mystr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.StringDocument"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Best Way\n",
    "sd2 = StringDocument(mystr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"samplefile.txt\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reading from a file\n",
    "filepath = \"samplefile.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.FileDocument"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Basic Way\n",
    "filedoc = Document(\"samplefile.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.FileDocument"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Best Way\n",
    "fd = FileDocument(\"samplefile.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### There is also \n",
    "+ TokenDocument()\n",
    "+ NGramDocument()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The best error message is the one that never shows up.\\nYou Learn More From Failure Than From Success. \\nThe purpose of software engineering is to control complexity, not to create it\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Working  With Our Document\n",
    "text(sd1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What language is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Languages.EnglishLanguage"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting the Base Info About it\n",
    "language(sd1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization With TextAnalysis\n",
    "+ Word Tokens\n",
    "+ Sentence Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The best error message is the one that never shows up.\\nYou Learn More From Failure Than From Success. \\nThe purpose of software engineering is to control complexity, not to create it\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text(sd1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32-element Array{SubString{String},1}:\n",
       " \"The\"        \n",
       " \"best\"       \n",
       " \"error\"      \n",
       " \"message\"    \n",
       " \"is\"         \n",
       " \"the\"        \n",
       " \"one\"        \n",
       " \"that\"       \n",
       " \"never\"      \n",
       " \"shows\"      \n",
       " \"up.\"        \n",
       " \"You\"        \n",
       " \"Learn\"      \n",
       " ⋮            \n",
       " \"purpose\"    \n",
       " \"of\"         \n",
       " \"software\"   \n",
       " \"engineering\"\n",
       " \"is\"         \n",
       " \"to\"         \n",
       " \"control\"    \n",
       " \"complexity,\"\n",
       " \"not\"        \n",
       " \"to\"         \n",
       " \"create\"     \n",
       " \"it\"         "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Word Tokens from a String Document\n",
    "tokens(sd1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The best error message is the one that never shows up.\\r\\nYou Learn More From Failure Than From Success. \\r\\nThe purpose of software engineering is to control complexity, not to create it\""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text(fd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32-element Array{SubString{String},1}:\n",
       " \"The\"        \n",
       " \"best\"       \n",
       " \"error\"      \n",
       " \"message\"    \n",
       " \"is\"         \n",
       " \"the\"        \n",
       " \"one\"        \n",
       " \"that\"       \n",
       " \"never\"      \n",
       " \"shows\"      \n",
       " \"up.\"        \n",
       " \"You\"        \n",
       " \"Learn\"      \n",
       " ⋮            \n",
       " \"purpose\"    \n",
       " \"of\"         \n",
       " \"software\"   \n",
       " \"engineering\"\n",
       " \"is\"         \n",
       " \"to\"         \n",
       " \"control\"    \n",
       " \"complexity,\"\n",
       " \"not\"        \n",
       " \"to\"         \n",
       " \"create\"     \n",
       " \"it\"         "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Word Tokens from a File Document\n",
    "tokens(fd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization With WordTokenizer\n",
    "+ Word Tokens\n",
    "+ Sentence Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "using WordTokenizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.StringDocument"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sd1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33-element Array{SubString{String},1}:\n",
       " \"The\"        \n",
       " \"best\"       \n",
       " \"error\"      \n",
       " \"message\"    \n",
       " \"is\"         \n",
       " \"the\"        \n",
       " \"one\"        \n",
       " \"that\"       \n",
       " \"never\"      \n",
       " \"shows\"      \n",
       " \"up.\"        \n",
       " \"You\"        \n",
       " \"Learn\"      \n",
       " ⋮            \n",
       " \"of\"         \n",
       " \"software\"   \n",
       " \"engineering\"\n",
       " \"is\"         \n",
       " \"to\"         \n",
       " \"control\"    \n",
       " \"complexity\" \n",
       " \",\"          \n",
       " \"not\"        \n",
       " \"to\"         \n",
       " \"create\"     \n",
       " \"it\"         "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Must convert from TextAnalysis Type to String Type\n",
    "tokenize(text(sd1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5-element Array{SubString{String},1}:\n",
       " \"Hello\"\n",
       " \"world\"\n",
       " \"this\" \n",
       " \"is\"   \n",
       " \"Julia\""
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenize(\"Hello world this is Julia\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sentence Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, solve the problem. Then, write the code. \n",
    "Fix the cause, not the symptom.\n",
    "Simplicity is the soul of efficiency. Good design adds value faster than it adds cost.\n",
    "In theory, theory and practice are the same. In practice, they’re not.\n",
    "There are two ways of constructing a software design.\n",
    "One way is to make it so simple that there are obviously no deficiencies.\n",
    "And the other way is to make it so complicated that there are no obvious deficiencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.FileDocument"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read a file with sentences\n",
    "sent_files = FileDocument(\"quotesfiles.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\ufeffFirst, solve the problem. Then, write the code.\\r\\nFix the cause, not the symptom.\\r\\nSimplicity is the soul of efficiency.\\r\\nGood design adds value faster than it adds cost.\\r\\nIn theory, theory and practice are the same. In practice, they’re not.\\r\\nThere are two ways of constructing a software design.\\r\\nOne way is to make it so simple that there are obviously no deficiencies.\\r\\nAnd the other way is to make it so complicated that there are no obvious deficiencies.\""
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text(sent_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17-element Array{SubString{String},1}:\n",
       " \"\\ufeffFirst, solve the problem.\"                                                       \n",
       " \"Then, write the code.\"                                                                 \n",
       " \"\"                                                                                      \n",
       " \"Fix the cause, not the symptom.\"                                                       \n",
       " \"\"                                                                                      \n",
       " \"Simplicity is the soul of efficiency.\"                                                 \n",
       " \"\"                                                                                      \n",
       " \"Good design adds value faster than it adds cost.\"                                      \n",
       " \"\"                                                                                      \n",
       " \"In theory, theory and practice are the same.\"                                          \n",
       " \"In practice, they’re not.\"                                                             \n",
       " \"\"                                                                                      \n",
       " \"There are two ways of constructing a software design.\"                                 \n",
       " \"\"                                                                                      \n",
       " \"One way is to make it so simple that there are obviously no deficiencies.\"             \n",
       " \"\"                                                                                      \n",
       " \"And the other way is to make it so complicated that there are no obvious deficiencies.\""
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sentence Tokenization\n",
    "split_sentences(text(sent_files))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "﻿First, solve the problem.\n",
      "Then, write the code.\n",
      "\n",
      "Fix the cause, not the symptom.\n",
      "\n",
      "Simplicity is the soul of efficiency.\n",
      "\n",
      "Good design adds value faster than it adds cost.\n",
      "\n",
      "In theory, theory and practice are the same.\n",
      "In practice, they’re not.\n",
      "\n",
      "There are two ways of constructing a software design.\n",
      "\n",
      "One way is to make it so simple that there are obviously no deficiencies.\n",
      "\n",
      "And the other way is to make it so complicated that there are no obvious deficiencies.\n"
     ]
    }
   ],
   "source": [
    "for sentence in split_sentences(text(sent_files))\n",
    "    println(sentence)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word token=> SubString{String}[\"\\ufeffFirst\", \",\", \"solve\", \"the\", \"problem\", \".\"]\n",
      "Word token=> SubString{String}[\"Then\", \",\", \"write\", \"the\", \"code\", \".\"]\n",
      "Word token=> SubString{String}[]\n",
      "Word token=> SubString{String}[\"Fix\", \"the\", \"cause\", \",\", \"not\", \"the\", \"symptom\", \".\"]\n",
      "Word token=> SubString{String}[]\n",
      "Word token=> SubString{String}[\"Simplicity\", \"is\", \"the\", \"soul\", \"of\", \"efficiency\", \".\"]\n",
      "Word token=> SubString{String}[]\n",
      "Word token=> SubString{String}[\"Good\", \"design\", \"adds\", \"value\", \"faster\", \"than\", \"it\", \"adds\", \"cost\", \".\"]\n",
      "Word token=> SubString{String}[]\n",
      "Word token=> SubString{String}[\"In\", \"theory\", \",\", \"theory\", \"and\", \"practice\", \"are\", \"the\", \"same\", \".\"]\n",
      "Word token=> SubString{String}[\"In\", \"practice\", \",\", \"they\", \"’\", \"re\", \"not\", \".\"]\n",
      "Word token=> SubString{String}[]\n",
      "Word token=> SubString{String}[\"There\", \"are\", \"two\", \"ways\", \"of\", \"constructing\", \"a\", \"software\", \"design\", \".\"]\n",
      "Word token=> SubString{String}[]\n",
      "Word token=> SubString{String}[\"One\", \"way\", \"is\", \"to\", \"make\", \"it\", \"so\", \"simple\", \"that\", \"there\", \"are\", \"obviously\", \"no\", \"deficiencies\", \".\"]\n",
      "Word token=> SubString{String}[]\n",
      "Word token=> SubString{String}[\"And\", \"the\", \"other\", \"way\", \"is\", \"to\", \"make\", \"it\", \"so\", \"complicated\", \"that\", \"there\", \"are\", \"no\", \"obvious\", \"deficiencies\", \".\"]\n"
     ]
    }
   ],
   "source": [
    "for sentence in split_sentences(text(sent_files))\n",
    "    wordtokens = tokenize(sentence)\n",
    "    println(\"Word token=> $wordtokens\")\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N-Grams\n",
    "+ Combinations of multiple words\n",
    "+ Useful for creating features during language modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The best error message is the one that never shows up.\\nYou Learn More From Failure Than From Success. \\nThe purpose of software engineering is to control complexity, not to create it\""
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mystr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.StringDocument"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sd3 = StringDocument(mystr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{SubString{String},Int64} with 28 entries:\n",
       "  \"engineering\" => 1\n",
       "  \"Learn\"       => 1\n",
       "  \"is\"          => 2\n",
       "  \"From\"        => 2\n",
       "  \"not\"         => 1\n",
       "  \"one\"         => 1\n",
       "  \"never\"       => 1\n",
       "  \"up.\"         => 1\n",
       "  \"complexity,\" => 1\n",
       "  \"create\"      => 1\n",
       "  \"software\"    => 1\n",
       "  \"that\"        => 1\n",
       "  \"it\"          => 1\n",
       "  \"You\"         => 1\n",
       "  \"Failure\"     => 1\n",
       "  \"best\"        => 1\n",
       "  \"shows\"       => 1\n",
       "  \"purpose\"     => 1\n",
       "  \"error\"       => 1\n",
       "  \"the\"         => 1\n",
       "  \"Success.\"    => 1\n",
       "  \"The\"         => 2\n",
       "  \"Than\"        => 1\n",
       "  \"of\"          => 1\n",
       "  \"More\"        => 1\n",
       "  ⋮             => ⋮"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Unigram\n",
    "ngrams(sd3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{AbstractString,Int64} with 59 entries:\n",
       "  \"that never\"     => 1\n",
       "  \"is to\"          => 1\n",
       "  \"create\"         => 1\n",
       "  \"that\"           => 1\n",
       "  \"best\"           => 1\n",
       "  \"Than From\"      => 1\n",
       "  \"shows up.\"      => 1\n",
       "  \"purpose\"        => 1\n",
       "  \"of\"             => 1\n",
       "  \"purpose of\"     => 1\n",
       "  \"More\"           => 1\n",
       "  \"to\"             => 2\n",
       "  \"the one\"        => 1\n",
       "  \"is\"             => 2\n",
       "  \"never\"          => 1\n",
       "  \"complexity,\"    => 1\n",
       "  \"software\"       => 1\n",
       "  \"one that\"       => 1\n",
       "  \"shows\"          => 1\n",
       "  \"From Success.\"  => 1\n",
       "  \"The purpose\"    => 1\n",
       "  \"message is\"     => 1\n",
       "  \"engineering is\" => 1\n",
       "  \"engineering\"    => 1\n",
       "  \"not\"            => 1\n",
       "  ⋮                => ⋮"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Bigrams\n",
    "ngrams(sd3,2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pair{AbstractString,Int64}(\"that never\", 1)\n",
      "Pair{AbstractString,Int64}(\"Success. The purpose\", 1)\n",
      "Pair{AbstractString,Int64}(\"You Learn More\", 1)\n",
      "Pair{AbstractString,Int64}(\"From Failure Than\", 1)\n",
      "Pair{AbstractString,Int64}(\"is to\", 1)\n",
      "Pair{AbstractString,Int64}(\"purpose of software\", 1)\n",
      "Pair{AbstractString,Int64}(\"create\", 1)\n",
      "Pair{AbstractString,Int64}(\"the one that\", 1)\n",
      "Pair{AbstractString,Int64}(\"software engineering is\", 1)\n",
      "Pair{AbstractString,Int64}(\"that\", 1)\n",
      "Pair{AbstractString,Int64}(\"control complexity, not\", 1)\n",
      "Pair{AbstractString,Int64}(\"best\", 1)\n",
      "Pair{AbstractString,Int64}(\"Than From\", 1)\n",
      "Pair{AbstractString,Int64}(\"shows up.\", 1)\n",
      "Pair{AbstractString,Int64}(\"purpose\", 1)\n",
      "Pair{AbstractString,Int64}(\"of\", 1)\n",
      "Pair{AbstractString,Int64}(\"purpose of\", 1)\n",
      "Pair{AbstractString,Int64}(\"More\", 1)\n",
      "Pair{AbstractString,Int64}(\"to\", 2)\n",
      "Pair{AbstractString,Int64}(\"the one\", 1)\n",
      "Pair{AbstractString,Int64}(\"is\", 2)\n",
      "Pair{AbstractString,Int64}(\"is the one\", 1)\n",
      "Pair{AbstractString,Int64}(\"one that never\", 1)\n",
      "Pair{AbstractString,Int64}(\"never\", 1)\n",
      "Pair{AbstractString,Int64}(\"complexity,\", 1)\n",
      "Pair{AbstractString,Int64}(\"message is the\", 1)\n",
      "Pair{AbstractString,Int64}(\"software\", 1)\n",
      "Pair{AbstractString,Int64}(\"error message is\", 1)\n",
      "Pair{AbstractString,Int64}(\"shows up. You\", 1)\n",
      "Pair{AbstractString,Int64}(\"one that\", 1)\n",
      "Pair{AbstractString,Int64}(\"shows\", 1)\n",
      "Pair{AbstractString,Int64}(\"that never shows\", 1)\n",
      "Pair{AbstractString,Int64}(\"From Success.\", 1)\n",
      "Pair{AbstractString,Int64}(\"The purpose\", 1)\n",
      "Pair{AbstractString,Int64}(\"to control complexity,\", 1)\n",
      "Pair{AbstractString,Int64}(\"message is\", 1)\n",
      "Pair{AbstractString,Int64}(\"engineering is\", 1)\n",
      "Pair{AbstractString,Int64}(\"engineering\", 1)\n",
      "Pair{AbstractString,Int64}(\"not\", 1)\n",
      "Pair{AbstractString,Int64}(\"best error message\", 1)\n",
      "Pair{AbstractString,Int64}(\"is to control\", 1)\n",
      "Pair{AbstractString,Int64}(\"not to create\", 1)\n",
      "Pair{AbstractString,Int64}(\"to create it\", 1)\n",
      "Pair{AbstractString,Int64}(\"The best\", 1)\n",
      "Pair{AbstractString,Int64}(\"Failure Than From\", 1)\n",
      "Pair{AbstractString,Int64}(\"software engineering\", 1)\n",
      "Pair{AbstractString,Int64}(\"best error\", 1)\n",
      "Pair{AbstractString,Int64}(\"More From\", 1)\n",
      "Pair{AbstractString,Int64}(\"From Failure\", 1)\n",
      "Pair{AbstractString,Int64}(\"not to\", 1)\n",
      "Pair{AbstractString,Int64}(\"Failure\", 1)\n",
      "Pair{AbstractString,Int64}(\"up. You\", 1)\n",
      "Pair{AbstractString,Int64}(\"of software\", 1)\n",
      "Pair{AbstractString,Int64}(\"error\", 1)\n",
      "Pair{AbstractString,Int64}(\"error message\", 1)\n",
      "Pair{AbstractString,Int64}(\"Failure Than\", 1)\n",
      "Pair{AbstractString,Int64}(\"Success. The\", 1)\n",
      "Pair{AbstractString,Int64}(\"The\", 2)\n",
      "Pair{AbstractString,Int64}(\"up. You Learn\", 1)\n",
      "Pair{AbstractString,Int64}(\"The purpose of\", 1)\n",
      "Pair{AbstractString,Int64}(\"complexity, not to\", 1)\n",
      "Pair{AbstractString,Int64}(\"message\", 1)\n",
      "Pair{AbstractString,Int64}(\"create it\", 1)\n",
      "Pair{AbstractString,Int64}(\"never shows up.\", 1)\n",
      "Pair{AbstractString,Int64}(\"control complexity,\", 1)\n",
      "Pair{AbstractString,Int64}(\"control\", 1)\n",
      "Pair{AbstractString,Int64}(\"The best error\", 1)\n",
      "Pair{AbstractString,Int64}(\"Learn\", 1)\n",
      "Pair{AbstractString,Int64}(\"From\", 2)\n",
      "Pair{AbstractString,Int64}(\"one\", 1)\n",
      "Pair{AbstractString,Int64}(\"up.\", 1)\n",
      "Pair{AbstractString,Int64}(\"Learn More From\", 1)\n",
      "Pair{AbstractString,Int64}(\"to create\", 1)\n",
      "Pair{AbstractString,Int64}(\"of software engineering\", 1)\n",
      "Pair{AbstractString,Int64}(\"Learn More\", 1)\n",
      "Pair{AbstractString,Int64}(\"it\", 1)\n",
      "Pair{AbstractString,Int64}(\"You\", 1)\n",
      "Pair{AbstractString,Int64}(\"You Learn\", 1)\n",
      "Pair{AbstractString,Int64}(\"to control\", 1)\n",
      "Pair{AbstractString,Int64}(\"the\", 1)\n",
      "Pair{AbstractString,Int64}(\"Success.\", 1)\n",
      "Pair{AbstractString,Int64}(\"More From Failure\", 1)\n",
      "Pair{AbstractString,Int64}(\"Than From Success.\", 1)\n",
      "Pair{AbstractString,Int64}(\"is the\", 1)\n",
      "Pair{AbstractString,Int64}(\"Than\", 1)\n",
      "Pair{AbstractString,Int64}(\"From Success. The\", 1)\n",
      "Pair{AbstractString,Int64}(\"complexity, not\", 1)\n",
      "Pair{AbstractString,Int64}(\"never shows\", 1)\n",
      "Pair{AbstractString,Int64}(\"engineering is to\", 1)\n"
     ]
    }
   ],
   "source": [
    "# Trigram\n",
    "for trigram in ngrams(sd3,3)\n",
    "    println(trigram)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String,Int64} with 6 entries:\n",
       "  \"or\"    => 1\n",
       "  \"be...\" => 1\n",
       "  \"not\"   => 1\n",
       "  \"to\"    => 1\n",
       "  \"To\"    => 1\n",
       "  \"be\"    => 2"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating an NGram \n",
    "my_ngrams = Dict{String, Int}(\"To\" => 1, \"be\" => 2,\n",
    "                                \"or\" => 1, \"not\" => 1,\n",
    "                                \"to\" => 1, \"be...\" => 1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.NGramDocument"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngd = NGramDocument(my_ngrams)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Detecting Which NGram it is\n",
    "ngram_complexity(ngd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{AbstractString,Int64} with 18 entries:\n",
       "  \"that never\"  => 1\n",
       "  \"shows up.\"   => 1\n",
       "  \"the one\"     => 1\n",
       "  \"purpose\"     => 1\n",
       "  \"is\"          => 2\n",
       "  \"never\"       => 1\n",
       "  \"complexity,\" => 1\n",
       "  \"is to\"       => 1\n",
       "  \"of\"          => 1\n",
       "  \"create\"      => 1\n",
       "  \"purpose of\"  => 1\n",
       "  \"software\"    => 1\n",
       "  \"that\"        => 1\n",
       "  \"More\"        => 1\n",
       "  \"one that\"    => 1\n",
       "  \"to\"          => 2\n",
       "  \"best\"        => 1\n",
       "  \"Than From\"   => 1"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_ngrams2 = Dict{AbstractString,Int64}(\n",
    "  \"that never\" => 1,\"is to\" => 1,\"create\" => 1,\"that\" => 1,\"best\" => 1,\"Than From\" => 1,\"shows up.\" => 1,\n",
    "    \"purpose\" => 1,\"of\" => 1,\"purpose of\" => 1,\"More\" => 1,\"to\" => 2,\"the one\" => 1,\n",
    "    \"is\" => 2,\"never\" => 1,\"complexity,\"=> 1,\"software\" => 1,\"one that\" => 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.NGramDocument"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngd2 = NGramDocument(my_ngrams2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngram_complexity(ngd2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You can use pip to install nltk and add it\n",
    "+ pip install nltk\n",
    "- import nltk\n",
    "- nltk.download\n",
    "\n",
    "#### In Julia\n",
    "+ using Conda\n",
    "+ Conda.add(\"nltk\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parts of Speech\n",
    "+ NLTK.tags via PyCall\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "using PyCall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing Part of Speech Tag from NLTK\n",
    "@pyimport nltk.tag as ptag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A TextAnalysis.StringDocument"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using TextAnalysis to tokenize or WordTokenizer to do the same\n",
    "ex = StringDocument(\"Julia is very fast but it is still young\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9-element Array{SubString{String},1}:\n",
       " \"Julia\"\n",
       " \"is\"   \n",
       " \"very\" \n",
       " \"fast\" \n",
       " \"but\"  \n",
       " \"it\"   \n",
       " \"is\"   \n",
       " \"still\"\n",
       " \"young\""
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TextAnalysis.tokens()\n",
    "mytokens = tokens(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9-element Array{Tuple{String,String},1}:\n",
       " (\"Julia\", \"NNP\")\n",
       " (\"is\", \"VBZ\")   \n",
       " (\"very\", \"RB\")  \n",
       " (\"fast\", \"RB\")  \n",
       " (\"but\", \"CC\")   \n",
       " (\"it\", \"PRP\")   \n",
       " (\"is\", \"VBZ\")   \n",
       " (\"still\", \"RB\") \n",
       " (\"young\", \"JJ\") "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using NLTK tags for finding the part of speech of our tokens\n",
    "ptag.pos_tag(mytokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Inflection == Word Formation by adding to base/root word\n",
    "+ Stemming (Basics) stem!()\n",
    "+ Lemmatizing\n",
    "- - How do we do these?\n",
    "- + PyCall To the Rescue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              AbstractDocument     92 bytes  DataType\n",
      "                        Corpus     40 bytes  UnionAll\n",
      "               DirectoryCorpus      0 bytes  TextAnalysis.#DirectoryCorpus\n",
      "                      Document      0 bytes  TextAnalysis.#Document\n",
      "            DocumentTermMatrix    136 bytes  DataType\n",
      "                  FileDocument    124 bytes  DataType\n",
      "               GenericDocument     48 bytes  Union\n",
      "                 NGramDocument    136 bytes  DataType\n",
      "                       Stemmer    136 bytes  DataType\n",
      "                StringDocument    124 bytes  DataType\n",
      "                  TextAnalysis  23658 KB     Module\n",
      "              TextHashFunction    124 bytes  DataType\n",
      "                 TokenDocument    124 bytes  DataType\n",
      "                        author      0 bytes  TextAnalysis.#author\n",
      "                       author!      0 bytes  TextAnalysis.#author!\n",
      "                   cardinality      0 bytes  TextAnalysis.#cardinality\n",
      "                     documents      0 bytes  TextAnalysis.#documents\n",
      "                           dtm      0 bytes  TextAnalysis.#dtm\n",
      "                           dtv      0 bytes  TextAnalysis.#dtv\n",
      "                      each_dtv      0 bytes  TextAnalysis.#each_dtv\n",
      "                 each_hash_dtv      0 bytes  TextAnalysis.#each_hash_dtv\n",
      "                frequent_terms      0 bytes  TextAnalysis.#frequent_terms\n",
      "                      hash_dtm      0 bytes  TextAnalysis.#hash_dtm\n",
      "                      hash_dtv      0 bytes  TextAnalysis.#hash_dtv\n",
      "                 hash_function      0 bytes  TextAnalysis.#hash_function\n",
      "                hash_function!      0 bytes  TextAnalysis.#hash_function!\n",
      "                      hash_tdm      0 bytes  TextAnalysis.#hash_tdm\n",
      "                    index_hash      0 bytes  TextAnalysis.#index_hash\n",
      "                    index_size      0 bytes  TextAnalysis.#index_size\n",
      "                 inverse_index      0 bytes  TextAnalysis.#inverse_index\n",
      "                      language      0 bytes  TextAnalysis.#language\n",
      "                     language!      0 bytes  TextAnalysis.#language!\n",
      "                           lda      0 bytes  TextAnalysis.#lda\n",
      "             lexical_frequency      0 bytes  TextAnalysis.#lexical_frequency\n",
      "                       lexicon      0 bytes  TextAnalysis.#lexicon\n",
      "                  lexicon_size      0 bytes  TextAnalysis.#lexicon_size\n",
      "                           lsa      0 bytes  TextAnalysis.#lsa\n",
      "                          name      0 bytes  Languages.#name\n",
      "                         name!      0 bytes  TextAnalysis.#name!\n",
      "              ngram_complexity      0 bytes  TextAnalysis.#ngram_complexity\n",
      "                        ngrams      0 bytes  TextAnalysis.#ngrams\n",
      "                       ngrams!      0 bytes  TextAnalysis.#ngrams!\n",
      "                      prepare!      0 bytes  TextAnalysis.#prepare!\n",
      "              remove_articles!      0 bytes  TextAnalysis.#remove_articles!\n",
      "                   remove_case      0 bytes  TextAnalysis.#remove_case\n",
      "                  remove_case!      0 bytes  TextAnalysis.#remove_case!\n",
      "           remove_corrupt_utf8      0 bytes  TextAnalysis.#remove_corrupt_utf8\n",
      "          remove_corrupt_utf8!      0 bytes  TextAnalysis.#remove_corrupt_utf8!\n",
      "     remove_definite_articles!      0 bytes  TextAnalysis.#remove_definite_arti…\n",
      "        remove_frequent_terms!      0 bytes  TextAnalysis.#remove_frequent_term…\n",
      "              remove_html_tags      0 bytes  TextAnalysis.#remove_html_tags\n",
      "             remove_html_tags!      0 bytes  TextAnalysis.#remove_html_tags!\n",
      "   remove_indefinite_articles!      0 bytes  TextAnalysis.#remove_indefinite_ar…\n",
      "            remove_nonletters!      0 bytes  TextAnalysis.#remove_nonletters!\n",
      "               remove_numbers!      0 bytes  TextAnalysis.#remove_numbers!\n",
      "               remove_patterns      0 bytes  TextAnalysis.#remove_patterns\n",
      "              remove_patterns!      0 bytes  TextAnalysis.#remove_patterns!\n",
      "          remove_prepositions!      0 bytes  TextAnalysis.#remove_prepositions!\n",
      "              remove_pronouns!      0 bytes  TextAnalysis.#remove_pronouns!\n",
      "           remove_punctuation!      0 bytes  TextAnalysis.#remove_punctuation!\n",
      "          remove_sparse_terms!      0 bytes  TextAnalysis.#remove_sparse_terms!\n",
      "            remove_stop_words!      0 bytes  TextAnalysis.#remove_stop_words!\n",
      "            remove_whitespace!      0 bytes  TextAnalysis.#remove_whitespace!\n",
      "                 remove_words!      0 bytes  TextAnalysis.#remove_words!\n",
      "                  sparse_terms      0 bytes  TextAnalysis.#sparse_terms\n",
      "                  standardize!      0 bytes  TextAnalysis.#standardize!\n",
      "                          stem      0 bytes  TextAnalysis.#stem\n",
      "                         stem!      0 bytes  TextAnalysis.#stem!\n",
      "                    stem_words      4 bytes  UInt32\n",
      "                 stemmer_types      0 bytes  TextAnalysis.#stemmer_types\n",
      "                strip_articles      4 bytes  UInt32\n",
      "                    strip_case      4 bytes  UInt32\n",
      "            strip_corrupt_utf8      4 bytes  UInt32\n",
      "       strip_definite_articles      4 bytes  UInt32\n",
      "          strip_frequent_terms      4 bytes  UInt32\n",
      "               strip_html_tags      4 bytes  UInt32\n",
      "     strip_indefinite_articles      4 bytes  UInt32\n",
      "             strip_non_letters      4 bytes  UInt32\n",
      "                 strip_numbers      4 bytes  UInt32\n",
      "                strip_patterns      4 bytes  UInt32\n",
      "            strip_prepositions      4 bytes  UInt32\n",
      "                strip_pronouns      4 bytes  UInt32\n",
      "             strip_punctuation      4 bytes  UInt32\n",
      "            strip_sparse_terms      4 bytes  UInt32\n",
      "               strip_stopwords      4 bytes  UInt32\n",
      "              strip_whitespace      4 bytes  UInt32\n",
      "            tag_part_of_speech      4 bytes  UInt32\n",
      "                      tag_pos!      0 bytes  TextAnalysis.#tag_pos!\n",
      "                           tdm      0 bytes  TextAnalysis.#tdm\n",
      "                          text      0 bytes  TextAnalysis.#text\n",
      "                         text!      0 bytes  TextAnalysis.#text!\n",
      "                            tf      0 bytes  TextAnalysis.#tf\n",
      "                           tf!      0 bytes  TextAnalysis.#tf!\n",
      "                        tf_idf      0 bytes  TextAnalysis.#tf_idf\n",
      "                       tf_idf!      0 bytes  TextAnalysis.#tf_idf!\n",
      "                     timestamp      0 bytes  TextAnalysis.#timestamp\n",
      "                    timestamp!      0 bytes  TextAnalysis.#timestamp!\n",
      "                        tokens      0 bytes  TextAnalysis.#tokens\n",
      "                       tokens!      0 bytes  TextAnalysis.#tokens!\n",
      "         update_inverse_index!      0 bytes  TextAnalysis.#update_inverse_index!\n",
      "               update_lexicon!      0 bytes  TextAnalysis.#update_lexicon!\n"
     ]
    }
   ],
   "source": [
    "whos(TextAnalysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Path to \n",
    "# /usr/local/lib/python3.5/dist-packages/textblob\n",
    "# /usr/local/lib/python3.5/dist-packages/nltk"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.6.2",
   "language": "julia",
   "name": "julia-0.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}