How to Find the Most Common Words Using Spacy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to Find the Most Common Words Using SpaCy & Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Our Packages\n",
    "import spacy\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "docx = nlp(open('luke6.txt').read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Blessings and Woes\n",
       "He went down with them and stood on a level place. A large crowd of his disciples was there and a great number of people from all over Judea, from Jerusalem, and from the coastal region around Tyre and Sidon, who had come to hear him and to be healed of their diseases. Those troubled by impure spirits were cured, and the people all tried to touch him, because power was coming from him and healing them all. Looking at his disciples, he said: \"Blessed are you who are poor, for yours is the kingdom of God. Blessed are you who hunger now, for you will be satisfied. Blessed are you who weep now, for you will laugh. Blessed are you when people hate you, when they exclude you and insult you and reject your name as evil, because of the Son of Man. \"Rejoice in that day and leap for joy, because great is your reward in heaven. For that is how their ancestors treated the prophets. \"But woe to you who are rich, for you have already received your comfort. Woe to you who are well fed now, for you will go hungry. Woe to you who laugh now, for you will mourn and weep. Woe to you when everyone speaks well of you, for that is how their ancestors treated the false prophets.\n",
       "Love for Enemies\n",
       "\"But to you who are listening I say: Love your enemies, do good to those who hate you, bless those who curse you, pray for those who mistreat you. If someone slaps you on one cheek, turn to them the other also. If someone takes your coat, do not withhold your shirt from them. Give to everyone who asks you, and if anyone takes what belongs to you, do not demand it back. Do to others as you would have them do to you. \"If you love those who love you, what credit is that to you? Even sinners love those who love them. And if you do good to those who are good to you, what credit is that to you? Even sinners do that. And if you lend to those from whom you expect repayment, what credit is that to you? Even sinners lend to sinners, expecting to be repaid in full. But love your enemies, do good to them, and lend to them without expecting to get anything back. Then your reward will be great, and you will be children of the Most High, because he is kind to the ungrateful and wicked. Be merciful, just as your Father is merciful.\n",
       "Judging Others\n",
       "\"Do not judge, and you will not be judged. Do not condemn, and you will not be condemned. Forgive, and you will be forgiven. Give, and it will be given to you. A good measure, pressed down, shaken together and running over, will be poured into your lap. For with the measure you use, it will be measured to you.\" He also told them this parable: \"Can the blind lead the blind? Will they not both fall into a pit? The student is not above the teacher, but everyone who is fully trained will be like their teacher. \"Why do you look at the speck of sawdust in your brotherâ€™s eye and pay no attention to the plank in your own eye? How can you say to your brother, 'Brother, let me take the speck out of your eye,'when you yourself fail to see the plank in your own eye? You hypocrite, first take the plank out of your eye, and then you will see clearly to remove the speck from your brotherâ€™s eye.\n",
       "A Tree and Its Fruit\n",
       "\"No good tree bears bad fruit, nor does a bad tree bear good fruit. Each tree is recognized by its own fruit. People do not pick figs from thornbushes, or grapes from briers. A good man brings good things out of the good stored up in his heart, and an evil man brings evil things out of the evil stored up in his heart. For the mouth speaks what the heart is full of.\n",
       "The Wise and Foolish Builders\n",
       "\"Why do you call me, 'Lord, Lord,'and do not do what I say? As for everyone who comes to me and hears my words and puts them into practice, I will show you what they are like. They are like a man building a house, who dug down deep and laid the foundation on rock. When a flood came, the torrent struck that house but could not shake it, because it was well built. But the one who hears my words and does not put them into practice is like a man who built a house on the ground without a foundation. The moment the torrent struck that house, it collapsed and its destruction was complete.\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove Punct,Stop \n",
    "# Nouns\n",
    "nouns = [ token.text for token in docx if token.is_stop != True and token.is_punct !=True and token.pos_ == 'NOUN']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Blessings',\n",
       " 'level',\n",
       " 'place',\n",
       " 'crowd',\n",
       " 'disciples',\n",
       " 'number',\n",
       " 'people',\n",
       " 'region',\n",
       " 'diseases',\n",
       " 'impure',\n",
       " 'spirits',\n",
       " 'people',\n",
       " 'power',\n",
       " 'disciples',\n",
       " 'kingdom',\n",
       " 'people',\n",
       " 'evil',\n",
       " 'day',\n",
       " 'leap',\n",
       " 'joy',\n",
       " 'reward',\n",
       " 'ancestors',\n",
       " 'prophets',\n",
       " 'woe',\n",
       " 'comfort',\n",
       " 'Woe',\n",
       " 'Woe',\n",
       " 'ancestors',\n",
       " 'prophets',\n",
       " 'enemies',\n",
       " 'good',\n",
       " 'cheek',\n",
       " 'coat',\n",
       " 'shirt',\n",
       " 'credit',\n",
       " 'sinners',\n",
       " 'good',\n",
       " 'credit',\n",
       " 'sinners',\n",
       " 'repayment',\n",
       " 'credit',\n",
       " 'sinners',\n",
       " 'sinners',\n",
       " 'enemies',\n",
       " 'good',\n",
       " 'reward',\n",
       " 'children',\n",
       " 'Others',\n",
       " 'measure',\n",
       " 'lap',\n",
       " 'measure',\n",
       " 'parable',\n",
       " 'pit',\n",
       " 'student',\n",
       " 'teacher',\n",
       " 'teacher',\n",
       " 'speck',\n",
       " 'sawdust',\n",
       " 'brotherâ€',\n",
       " 'eye',\n",
       " 'attention',\n",
       " 'plank',\n",
       " 'eye',\n",
       " 'brother',\n",
       " 'Brother',\n",
       " 'speck',\n",
       " 'plank',\n",
       " 'eye',\n",
       " 'plank',\n",
       " 'eye',\n",
       " 'speck',\n",
       " 'brotherâ€',\n",
       " 'eye',\n",
       " 'tree',\n",
       " 'fruit',\n",
       " 'tree',\n",
       " 'fruit',\n",
       " 'tree',\n",
       " 'fruit',\n",
       " 'People',\n",
       " 'figs',\n",
       " 'thornbushes',\n",
       " 'grapes',\n",
       " 'briers',\n",
       " 'man',\n",
       " 'things',\n",
       " 'good',\n",
       " 'heart',\n",
       " 'man',\n",
       " 'things',\n",
       " 'evil',\n",
       " 'heart',\n",
       " 'mouth',\n",
       " 'heart',\n",
       " 'words',\n",
       " 'practice',\n",
       " 'man',\n",
       " 'house',\n",
       " 'foundation',\n",
       " 'rock',\n",
       " 'flood',\n",
       " 'torrent',\n",
       " 'house',\n",
       " 'words',\n",
       " 'practice',\n",
       " 'man',\n",
       " 'house',\n",
       " 'ground',\n",
       " 'foundation',\n",
       " 'moment',\n",
       " 'torrent',\n",
       " 'house',\n",
       " 'destruction']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nouns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_freq = Counter(nouns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "common_nouns = word_freq.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('eye', 5), ('good', 4), ('sinners', 4), ('man', 4), ('house', 4), ('people', 3), ('credit', 3), ('speck', 3), ('plank', 3), ('tree', 3)]\n"
     ]
    }
   ],
   "source": [
    "print(common_nouns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Most Common Verbs\n",
    "+ Some stops words can also be verbs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove Punct,Stop \n",
    "# verbs\n",
    "verbs = [ token.text for token in docx if token.is_punct !=True and token.pos_ == 'VERB']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('will', 15), ('is', 14), ('be', 12), ('do', 12), ('are', 11), ('love', 5), ('was', 4), ('Blessed', 4), ('say', 3), ('Do', 3)]\n"
     ]
    }
   ],
   "source": [
    "print(Counter(verbs).most_common(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "verbs_with_stopword = [ token.text for token in docx if token.is_stop != True and token.is_punct !=True and token.pos_ == 'VERB']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('love', 5), ('Blessed', 4), ('Do', 3), ('lend', 3), ('weep', 2), ('laugh', 2), ('hate', 2), ('treated', 2), ('speaks', 2), ('takes', 2)]\n"
     ]
    }
   ],
   "source": [
    "print(Counter(verbs_with_stopword).most_common(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}