"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"displacy.render(docx3,style='dep',options=options,jupyter=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Rendering HTML\n",
"+ Default is svg\n",
"+ set page to True\n",
"+ minify=True For Minified format"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"html = displacy.render(docx3,style='dep',page=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exporting The Rendered Graphic"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
"svg = displacy.render(doc, style='dep')\n",
"output = 'buffalosentence.svg'\n",
"with open(output,'w', encoding='utf-8') as f:\n",
" f.write(svg)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Alternative Method\n",
"# svg = displacy.render(doc, style='dep')\n",
"# output_path = Path('/images/sentence.svg')\n",
"# output_path.open('w', encoding='utf-8').write(svg)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Named Entity Recognition or Detection\n",
"+ Classifying a text into predefined categories or real world object entities.\n",
"+ takes a string of text (sentence or paragraph) as input and identifies relevant nouns (people, places, and organizations) that are mentioned in that string.\n",
"\n",
"##### Uses\n",
"+ Classifying or Categorizing contents by getting the relevant tags\n",
"+ Improve search algorithms\n",
"+ For content recommendations\n",
"+ For info extraction\n",
"\n",
"+ .ents\n",
"+ .label_"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"wikitext = nlp(u\"By 2020 the telecom company Orange, will relocate from Turkey to Orange County in the U.S. close to Apple.It will cost them 2 billion dollars.\")"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2020 DATE\n",
"Turkey GPE\n",
"Orange County GPE\n",
"U.S. GPE\n",
"Apple ORG\n",
"2 billion dollars MONEY\n"
]
}
],
"source": [
"for entity in wikitext.ents:\n",
" print(entity.text,entity.label_)"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Countries, cities, states'"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# What does GPE means\n",
"spacy.explain('GPE')"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"By \n",
"\n",
" 2020\n",
" DATE\n",
"\n",
" the telecom company Orange, will relocate from \n",
"\n",
" Turkey\n",
" GPE\n",
"\n",
" to \n",
"\n",
" Orange County\n",
" GPE\n",
"\n",
" in the \n",
"\n",
" U.S.\n",
" GPE\n",
"\n",
" close to \n",
"\n",
" Apple\n",
" ORG\n",
"\n",
".It will cost them \n",
"\n",
" 2 billion dollars\n",
" MONEY\n",
"\n",
".
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Visualize With DiSplaCy\n",
"displacy.render(wikitext,style='ent',jupyter=True)"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"wikitext2 = nlp(u\"Linus Benedict Torvalds is a Finnish-American software engineer who is the creator, and for a long time, principal developer of the Linux kernel, which became the kernel for operating systems such as the Linux operating systems, Android, and Chrome OS.\")"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" Linus Benedict Torvalds\n",
" PERSON\n",
"\n",
" is a \n",
"\n",
" Finnish\n",
" NORP\n",
"\n",
"-American software engineer who is the creator, and for a long time, principal developer of the \n",
"\n",
" Linux\n",
" ORG\n",
"\n",
" kernel, which became the kernel for operating systems such as the \n",
"\n",
" Linux\n",
" ORG\n",
"\n",
" operating systems, \n",
"\n",
" Android\n",
" GPE\n",
"\n",
", and Chrome OS.
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Visualize With DiSplaCy\n",
"displacy.render(wikitext2,style='ent',jupyter=True)"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Nationalities or religious or political groups'"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"spacy.explain('NORP')"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": [
"doc1 = nlp(\"Facebook, Explosion.ai, JCharisTech are all internet companies\")"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" Facebook\n",
" ORG\n",
"\n",
", \n",
"\n",
" Explosion.ai\n",
" ORG\n",
"\n",
", \n",
"\n",
" JCharisTech\n",
" ORG\n",
"\n",
" are all internet companies
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Visualize With DiSplaCy\n",
"displacy.render(doc1,style='ent',jupyter=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Text Normalization and Word Inflection\n",
"+ Word inflection == syntactic differences between word forms \n",
"+ Reducing a word to its base/root form\n",
"+ Lemmatization **\n",
"- - a word based on its intended meaning\n",
"+ Stemming \n",
"- - Cutting of the prefixes/suffices to reduce a word to base form\n",
"+ Word Shape Analysis"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [],
"source": [
"## Lemmatization \n",
"docx_lemma = nlp(\"studying student study studies studio studious\")"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Token=> studying Lemma=> study VERB\n",
"Token=> student Lemma=> student NOUN\n",
"Token=> study Lemma=> study NOUN\n",
"Token=> studies Lemma=> study NOUN\n",
"Token=> studio Lemma=> studio NOUN\n",
"Token=> studious Lemma=> studious ADJ\n"
]
}
],
"source": [
"for word in docx_lemma:\n",
" print(\"Token=>\",word.text,\"Lemma=>\",word.lemma_,word.pos_)"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [],
"source": [
"docx_lemma1 = nlp(\"good goods run running runner was be were\")"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Token=> good Lemma=> good ADJ\n",
"Token=> goods Lemma=> good NOUN\n",
"Token=> run Lemma=> run VERB\n",
"Token=> running Lemma=> run VERB\n",
"Token=> runner Lemma=> runner NOUN\n",
"Token=> was Lemma=> be VERB\n",
"Token=> be Lemma=> be VERB\n",
"Token=> were Lemma=> be VERB\n"
]
}
],
"source": [
"for word in docx_lemma1:\n",
" print(\"Token=>\",word.text,\"Lemma=>\",word.lemma_,word.pos_)"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [],
"source": [
"docx_lemma2 = nlp(\"walking walks walk walker\")"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Token=> walking Lemma=> walk VERB\n",
"Token=> walks Lemma=> walk NOUN\n",
"Token=> walk Lemma=> walk VERB\n",
"Token=> walker Lemma=> walker ADV\n"
]
}
],
"source": [
"for word in docx_lemma2:\n",
" print(\"Token=>\",word.text,\"Lemma=>\",word.lemma_,word.pos_)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Semantic Similarity\n",
"+ object1.similarity(object2)\n",
"+ Uses:\n",
"+ - Recommendation systems\n",
"+ - Data Preprocessing eg removing duplicates\n",
"- - python -m spacy download en_core_web_lg"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [],
"source": [
"# Loading Packages\n",
"import spacy\n",
"nlp = spacy.load('en')"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [],
"source": [
"# Similarity of object\n",
"doc1 = nlp(\"wolf\")\n",
"doc2 = nlp(\"dog\")"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.6759108958707175"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"doc1.similarity(doc2)"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [],
"source": [
"doc3 = nlp(\"cat\")"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.7344887997583573"
]
},
"execution_count": 64,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"doc3.similarity(doc2)"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [],
"source": [
"# Synonmys\n",
"doc4 = nlp(\"smart\")\n",
"doc5 = nlp(\"clever\")"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8051825859624082"
]
},
"execution_count": 66,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Similarity of words\n",
"doc4.similarity(doc5)"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [],
"source": [
"similarword = nlp(\"wolf dog cat bird fish\")"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"wolf\n",
"dog\n",
"cat\n",
"bird\n",
"fish\n"
]
}
],
"source": [
"for token in similarword:\n",
" print(token.text)"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"('wolf', 'wolf') similarity=> 1.0\n",
"('wolf', 'dog') similarity=> 0.5234999\n",
"('wolf', 'cat') similarity=> 0.30953428\n",
"('wolf', 'bird') similarity=> 0.52796596\n",
"('wolf', 'fish') similarity=> 0.051317338\n",
"('dog', 'wolf') similarity=> 0.5234999\n",
"('dog', 'dog') similarity=> 1.0\n",
"('dog', 'cat') similarity=> 0.62507194\n",
"('dog', 'bird') similarity=> 0.4794655\n",
"('dog', 'fish') similarity=> 0.32915178\n",
"('cat', 'wolf') similarity=> 0.30953428\n",
"('cat', 'dog') similarity=> 0.62507194\n",
"('cat', 'cat') similarity=> 1.0\n",
"('cat', 'bird') similarity=> 0.4474155\n",
"('cat', 'fish') similarity=> 0.447517\n",
"('bird', 'wolf') similarity=> 0.52796596\n",
"('bird', 'dog') similarity=> 0.4794655\n",
"('bird', 'cat') similarity=> 0.4474155\n",
"('bird', 'bird') similarity=> 1.0\n",
"('bird', 'fish') similarity=> 0.35412988\n",
"('fish', 'wolf') similarity=> 0.051317338\n",
"('fish', 'dog') similarity=> 0.32915178\n",
"('fish', 'cat') similarity=> 0.447517\n",
"('fish', 'bird') similarity=> 0.35412988\n",
"('fish', 'fish') similarity=> 1.0\n"
]
}
],
"source": [
"# Similarity Between Tokens\n",
"for token1 in similarword:\n",
" for token2 in similarword:\n",
" print((token1.text,token2.text),\"similarity=>\",token1.similarity(token2))"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {},
"outputs": [],
"source": [
"#[x for b in a for x in b] \n",
"mylist = [(token1.text,token2.text,token1.similarity(token2)) for token2 in similarword for token1 in similarword]"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('wolf', 'wolf', 1.0),\n",
" ('dog', 'wolf', 0.5234999),\n",
" ('cat', 'wolf', 0.30953428),\n",
" ('bird', 'wolf', 0.52796596),\n",
" ('fish', 'wolf', 0.051317338),\n",
" ('wolf', 'dog', 0.5234999),\n",
" ('dog', 'dog', 1.0),\n",
" ('cat', 'dog', 0.62507194),\n",
" ('bird', 'dog', 0.4794655),\n",
" ('fish', 'dog', 0.32915178),\n",
" ('wolf', 'cat', 0.30953428),\n",
" ('dog', 'cat', 0.62507194),\n",
" ('cat', 'cat', 1.0),\n",
" ('bird', 'cat', 0.4474155),\n",
" ('fish', 'cat', 0.447517),\n",
" ('wolf', 'bird', 0.52796596),\n",
" ('dog', 'bird', 0.4794655),\n",
" ('cat', 'bird', 0.4474155),\n",
" ('bird', 'bird', 1.0),\n",
" ('fish', 'bird', 0.35412988),\n",
" ('wolf', 'fish', 0.051317338),\n",
" ('dog', 'fish', 0.32915178),\n",
" ('cat', 'fish', 0.447517),\n",
" ('bird', 'fish', 0.35412988),\n",
" ('fish', 'fish', 1.0)]"
]
},
"execution_count": 71,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mylist"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using DataFrames"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {},
"outputs": [],
"source": [
"df = pd.DataFrame(mylist)"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" 0 | \n",
" 1 | \n",
" 2 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" wolf | \n",
" wolf | \n",
" 1.000000 | \n",
"
\n",
" \n",
" | 1 | \n",
" dog | \n",
" wolf | \n",
" 0.523500 | \n",
"
\n",
" \n",
" | 2 | \n",
" cat | \n",
" wolf | \n",
" 0.309534 | \n",
"
\n",
" \n",
" | 3 | \n",
" bird | \n",
" wolf | \n",
" 0.527966 | \n",
"
\n",
" \n",
" | 4 | \n",
" fish | \n",
" wolf | \n",
" 0.051317 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" 2 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2 | \n",
" 1.0 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Token1 | \n",
" Token2 | \n",
" Similarity | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" wolf | \n",
" wolf | \n",
" 1.000000 | \n",
"
\n",
" \n",
" | 1 | \n",
" dog | \n",
" wolf | \n",
" 0.523500 | \n",
"
\n",
" \n",
" | 2 | \n",
" cat | \n",
" wolf | \n",
" 0.309534 | \n",
"
\n",
" \n",
" | 3 | \n",
" bird | \n",
" wolf | \n",
" 0.527966 | \n",
"
\n",
" \n",
" | 4 | \n",
" fish | \n",
" wolf | \n",
" 0.051317 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Token1 | \n",
" Token2 | \n",
" Similarity | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0 | \n",
" 0 | \n",
" 1.000000 | \n",
"
\n",
" \n",
" | 1 | \n",
" 1 | \n",
" 0 | \n",
" 0.523500 | \n",
"
\n",
" \n",
" | 2 | \n",
" 2 | \n",
" 0 | \n",
" 0.309534 | \n",
"
\n",
" \n",
" | 3 | \n",
" 4 | \n",
" 0 | \n",
" 0.527966 | \n",
"
\n",
" \n",
" | 4 | \n",
" 3 | \n",
" 0 | \n",
" 0.051317 | \n",
"
\n",
" \n",
"
\n",
"