1-Day1-Tokenizing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "<h1>NLTK- Natural Language Toolkit </h1>\r\n",
    "\r\n",
    "<h2>NLTK: A tool for NLP\r\n",
    "NLP: Process of converting the natural languages to computer understandable data.\r\n",
    "\r\n",
    "NLTK documentation link: https://www.nltk.org/\r\n",
    "\r\n",
    "*To install nltk:*\r\n",
    "pip3 install nltk\r\n",
    "\r\n",
    "import nltk and download the nltk packages</h2>"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import nltk"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "nltk.download()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Once the popup opens, select \"all\" packages and download. After downloading finishes, click \"finish\""
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Tokenising: It's a form of grouping things\n",
    "Basic types of tokenisers: Word Tokeniser and Sentence Tokeniser\n",
    "\n",
    "Lexicons: words and their meanings - gets tricky with semantic analysis\n",
    "\n",
    "Corporas: body of text"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "from nltk.tokenize import word_tokenize, sent_tokenize"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "example_text = \"First Sentence. Second Sentence. Mr. Three's Sentence. Who's Mr. Four?\""
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "print(sent_tokenize(example_text))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['First Sentence.', 'Second Sentence.', \"Mr. Three's Sentence.\", \"Who's Mr. Four?\"]\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "print(word_tokenize(example_text))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['First', 'Sentence', '.', 'Second', 'Sentence', '.', 'Mr.', 'Three', \"'s\", 'Sentence', '.', 'Who', \"'s\", 'Mr.', 'Four', '?']\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "for x in word_tokenize(example_text): print(x)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "First\n",
      "Sentence\n",
      ".\n",
      "Second\n",
      "Sentence\n",
      ".\n",
      "Mr.\n",
      "Three\n",
      "'s\n",
      "Sentence\n",
      ".\n",
      "Who\n",
      "'s\n",
      "Mr.\n",
      "Four\n",
      "?\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.6.10 64-bit ('tf-gpu': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  },
  "interpreter": {
   "hash": "135e78ef6267b613ce7b86630936d470174b66187aad9f784a45e5cc3235687c"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}