FINAL. wikiscrape package.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# wikiscrape package (Save wikiscrape.py in repository)\n",
    "## This Python code can be used to search for a Wikipedia Article and do text analytics on it\n",
    "<br>\n",
    "To save a wikiscrape object, simply type: <br>\n",
    "<b> import wikiscrape </b><br>\n",
    "<b> var = wikiscrape.wiki('Search Article') </b> <br> <br>\n",
    "e.g. paris = wikiscrape.wiki('PArIs','Yes','french','Yes','Yes') means to search for the article Paris, auto format to proper case (Yes for 2nd argument, default Yes), search for French wikipedia (french for 3rd argument, default English), apply nltk stoplist for french (Yes for 4th argument, default No) and lemmatize the article using nltk (Yes for 5th argument, default No) <br>\n",
    "\n",
    "### Full capabilities of wikiscrape package include: <br>\n",
    "1. Able to search in multiple languages <br>\n",
    "2. Give suggestions on search terms if search is ambiguous <br>\n",
    "3. Gives a short summary (2 paragraphs) of the article if it is retrieved successfully <br>\n",
    "4. Retrieve full text or exact number of paragraphs in string output for data pipeline <br>\n",
    "5. var.HELP() for the full list of functions available <br>\n",
    "6. Basic error handling, including checking data type of arguments and reverting to defaults if errorneous args are given <br>\n",
    "7. Apply local stoplist by directly editing the dictionary, and also applying NLTK stoplist for multiple languages <br>\n",
    "8. Lemmatizing the article before using the text analytics functions <br>\n",
    "\n",
    "### Text Analytics capabilities include: <br>\n",
    "1. A frequency counter on the most common words in the Wikipedia article (after omitting common English words, or stoplist from NLTK for foreign languages). Can also find the Nth % of most common words, where 0 =< N =< 100. <br>\n",
    "2. A graph plot of the most common words in the Wikipedia article, and save the graph as an image <br>\n",
    "3. A graph plot on the most frequent Years mentioned in the article, to understand the Years of interest of the article, and save the graph as an image <br>\n",
    "4. A summary on the total number of words and total number of unique words after implementing the stoplist & lemmatization of common words. <br>\n",
    "5. Analytics functions available in wikiscrape object are commonwords, commonwordspct, plotwords, plotyear, totalwords, summary, gettext. \n",
    "<br><br>\n",
    "Refer to images in the repository for examples. The earliest image 'bar.png' made 4 months ago was the initial design for the bar chart for word frequency. Examples of the newest images (last edit: 27 Nov 2019) are 'ColdplayWordCount2.png' and 'Donald_Trump_40words.png'.\n",
    "<br><br>\n",
    "\n",
    "#### Libraries used: requests, bs4, collections, matplotlib, re, os, math, datetime, nltk (optional, only if using stoplist or lemmatization)\n",
    "Refer to requirements.txt <br>\n",
    "Package itself already has a comprehensive stoplist built inside to remove common words before text analytics <br>\n",
    "\n",
    "#### Updates:\n",
    "1. <b>26 May 2019</b> - Added plotyear() function to plot the most frequent years mentioned, and removed years in the frequency count of word counter (commonwords & commonwordspct functions).\n",
    "2. <b>9 June 2019</b> - Added markdown for explanation and added comments in the code for understanding <br>\n",
    "3. <b>13 June 2019</b> - Updated documentation for plotyear, plotwords, summary and gettext function in .HELP(). <br>\n",
    "4. <b>25 November 2019</b> - Update coming very soon to patch issues and improve on Wikipedia package, stay tuned! <br>\n",
    "5. <b>27 November 2019</b> - Major update to the Python package, including: <br>\n",
    "  a. Adding of lemmatization feature (using NLTK) before using text analytics functions <br>\n",
    "  b. Better documentation via docstrings and updating HELP function <br>\n",
    "  c. Improving of graph plotting design and font size for plotwords and plotyear <br>\n",
    "  d. Fixed some bugs for graph plotting including values not showing or showing up erroneously <br>\n",
    "  e. Refactored the code, provided better names for key variables for user understanding <br>\n",
    "  f. Performance improvement of article search by removing unused variables and functions <br>\n",
    "  g. Tested all functions and also error handling in case user puts in wrong parameters <br>\n",
    "6. <b>09 May 2020</b> - New feature to exclude N number of latest years in plotwords (e.g. from current year 2020, 2019, ...) and made graph titles larger\n",
    "7. For any questions or suggestions, please contact me at my Linkedin account - https://www.linkedin.com/in/kohjiaxuan/ <br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests # Get the HTML code\n",
    "from bs4 import BeautifulSoup # Tidy up the code\n",
    "from collections import Counter # Counter to count occurances of each word\n",
    "import matplotlib.pyplot as plt # graph plotting\n",
    "import re # regular expression to check if language setting is exactly 2 letters (for non common langs) in the argument\n",
    "import os # for plotwords to tell where file is saved\n",
    "import math # for calculating font size of graphs using exponential\n",
    "import datetime # for getting current year\n",
    "\n",
    "#var = wikiscrape.wiki('Article Search',optional arguments 2-4)\n",
    "#Arg 1 is article name in string, Arg 2 is to format in proper case (default Yes), Arg 3 is language (default EN), Arg 4 is use stoplist of NLTK (default No)\n",
    "class wiki:\n",
    "\n",
    "    #The main features of cleaning the wiki site and whether the site is valid is run in __init__\n",
    "    def __init__(self,title,option='Yes',lang='en',checknltk='No',lemmatize='No'):\n",
    "        \"\"\"The wiki() class accepts 5 arguments. The first one is a compulsory title of the Wikipedia page. \n",
    "        Second is to format the search string to proper/title case (Yes/No, default: Yes).\n",
    "        Third is for language settings (e.g. English, de, francais, etc., default: English). \n",
    "        Fourth is for implementing NLTK stoplist in provided languages (Yes/No, default: No, with standard stoplist)\n",
    "        By default, a standard stoplist of the most common words in the English language and Wikipedia common words is provided.\n",
    "        Fifth is for lemmatizing text before using text analytics functions (Yes/No, default: No).\n",
    "        \"\"\"\n",
    "        print(\"Page is loading...\\n\")\n",
    "        self.graphtitle = title #for graph title labelling\n",
    "        if isinstance(title, str) == True:\n",
    "            if str(option).lower() == 'yes' or str(option) == '':\n",
    "                self.title = str(title.title()) \n",
    "                print(\"Search text formatted to title/proper case by default. Set second argument as 'No' to disable formatting\")\n",
    "            elif str(option).lower() == 'no':\n",
    "                print(\"Search text has preserved the cases of each letter. Set second argument as 'Yes' to format to title/proper case\")\n",
    "                self.title = title\n",
    "            else:\n",
    "                self.title = title.title() \n",
    "                print('Invalid option for preserving case of search text, title/proper case will be used by default')\n",
    "            self.title = str(self.title.replace(\" \", \"_\")) #Convert spaces to _ as is Wikipedia page format\n",
    "        else:\n",
    "            print('Error encountered, search text (first argument) is not written as a string with quotes. Please try again')\n",
    "        \n",
    "        \n",
    "        #Checking if you should use NLTK library for stoplist, default set to False\n",
    "        self.nltkrun = False \n",
    "        if isinstance(checknltk, str): #check for string yes, no and other permutations\n",
    "            if checknltk.lower().strip() in {'yes','true','y','t'}:\n",
    "                try:\n",
    "                    import nltk\n",
    "                    from nltk.corpus import stopwords\n",
    "                    from nltk.corpus import wordnet\n",
    "                    self.nltkrun = True\n",
    "                except:\n",
    "                    print(\"stopwords and wordnet are not downloaded. To download, execute pip install nltk. Next, input nltk.download('stopwords') and nltk.download('wordnet')\")\n",
    "                    # nltk.download('stopwords')\n",
    "                    # nltk.download('wordnet')\n",
    "                    self.nltkrun = False\n",
    "        elif isinstance(checknltk, bool): #check for boolean yes/no\n",
    "            if checknltk == True:\n",
    "                try:\n",
    "                    import nltk\n",
    "                    from nltk.corpus import stopwords\n",
    "                    from nltk.corpus import wordnet\n",
    "                    self.nltkrun = True\n",
    "                except:\n",
    "                    print(\"stopwords and wordnet are not downloaded. To download, execute pip install nltk. Next, input nltk.download('stopwords') and nltk.download('wordnet')\")\n",
    "                    # nltk.download('stopwords')\n",
    "                    # nltk.download('wordnet')\n",
    "                    self.nltkrun = False\n",
    "            else:\n",
    "                self.nltkrun = False\n",
    "        else: #run default if options are invalid - don't run nltk stoplist\n",
    "            self.nltkrun = False\n",
    "            \n",
    "        # Check if NLTK Lemmatized is turned on\n",
    "        self.to_lemmatize = False\n",
    "        if isinstance(lemmatize, str):\n",
    "            if lemmatize.lower().strip() in {'yes','true','y','t'}:\n",
    "                from nltk.stem import WordNetLemmatizer\n",
    "                self.lemmatizer = WordNetLemmatizer()\n",
    "                self.to_lemmatize = True\n",
    "                print('Lemmatizing of Wikipedia text is enabled!')\n",
    "        elif isinstance(lemmatize, bool):\n",
    "            if lemmatize == True:\n",
    "                from nltk.stem import WordNetLemmatizer\n",
    "                self.lemmatizer = WordNetLemmatizer()\n",
    "                self.to_lemmatize = True\n",
    "                print('Lemmatizing of Wikipedia text is enabled!')\n",
    "\n",
    "        #Default: Stopword list obtained from nltk\n",
    "        self.nltkstopword = []\n",
    "        \n",
    "        #Detect language settings in third argument\n",
    "        self.lang = 'en'  \n",
    "        if isinstance(lang, str) == True:\n",
    "            if lang.lower().lstrip() in {'en','england','uk','u.k.','english','eng'}:\n",
    "                self.lang = 'en'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('english') \n",
    "                    print('NLTK Stopword list for English will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'de','germany','german','deutschland','deutsche','deutsch'}:\n",
    "                self.lang = 'de'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('german')\n",
    "                    print('NLTK Stopword list for German will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'ru','russian','russia','русский','российский','русская','россия'}:\n",
    "                self.lang = 'ru'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('russian')\n",
    "                    print('NLTK Stopword list for Russian will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'it','italy','italian','italia','italiano'}:\n",
    "                self.lang = 'it'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('italian')\n",
    "                    print('NLTK Stopword list for Italian will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'pt','por','portugal','portuguese','portogallo','portoghese','brazil','brasile','brasiliano','brazilian'}:\n",
    "                self.lang = 'pt'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('portuguese')\n",
    "                    print('NLTK Stopword list for Portuguese will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'ja','japan','japanese','jp','日本','ジャパン','日本語','にほん','にほんご'}:\n",
    "                self.lang = 'ja'\n",
    "            elif lang.lower().lstrip() in {'es','sp','spain','spanish','españa','espana','español','español','castellano','hispano'}:\n",
    "                self.lang = 'es'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('spanish')\n",
    "                    print('NLTK Stopword list for Spanish will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'fr','france','french','français','francais'}:\n",
    "                self.lang = 'fr'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('french')\n",
    "                    print('NLTK Stopword list for French will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'zh','cn','china','chinese','中文','华文','華文','汉语','漢語','中国','中國','漢語','华语','華語'}:\n",
    "                self.lang = 'zh'\n",
    "            elif lang.lower().lstrip() in {'pl','po','polish','poland','polska','polskie','polski'}:\n",
    "                self.lang = 'pl'\n",
    "            elif lang.lower().lstrip() in {'nl','netherlands','dutch','the netherlands','nederland','nederlands','holland','hollands'}:\n",
    "                self.lang = 'nl'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('dutch')\n",
    "                    print('NLTK Stopword list for Dutch will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'sv','sw','sweden','swedish','sverige','svenska','svensk'}:\n",
    "                self.lang = 'sv'\n",
    "                if self.nltkrun == True:\n",
    "                    self.nltkstopword = stopwords.words('swedish')\n",
    "                    print('NLTK Stopword list for Swedish will be used to remove common words\\n')\n",
    "            elif lang.lower().lstrip() in {'vi','vietnam','vn','vietnamese','việt nam','việtnam','tiếng việt','tiếng Việt'}:\n",
    "                self.lang = 'vi'\n",
    "            elif re.search(r\"[a-z]{2}\", lang, re.I) != None: #search for two letter character as lang set, not case sensitive re.I\n",
    "            #elif len((lang.lower()).lstrip()) == 2:\n",
    "                self.lang = lang\n",
    "                print('Language set to ' + \"'\" + lang + \"'\")\n",
    "            else:\n",
    "                self.lang = 'en'\n",
    "                print('Invalid language settings, English articles searched by default\\n')\n",
    "                \n",
    "        else:\n",
    "            self.lang = 'en'\n",
    "            print('Invalid language settings or not in string format, hence English articles used by default\\n')\n",
    "        \n",
    "        #Get URL of Wikipedia Article\n",
    "        self.url = 'https://' + self.lang + '.wikipedia.org/wiki/' + self.title #combine the two to get full URL\n",
    "\n",
    "        try: \n",
    "            self.page = requests.get(self.url) #retrieve HTML info from site\n",
    "        except:\n",
    "            self.lang = 'en'\n",
    "            self.url = 'https://' + self.lang + '.wikipedia.org/wiki/' + self.title\n",
    "            self.page = requests.get(self.url)\n",
    "            print('Error with language settings, English used as default\\n')\n",
    "\n",
    "        self.contents = self.page.content \n",
    "        #Parse the HTML nicely with formatting\n",
    "        self.soup = BeautifulSoup(self.contents, 'html.parser') \n",
    "        \n",
    "        self.wordcorpus = self.soup.find_all('p') #obtain all paragraphs starting with tag <p>\n",
    "        self.wordcorpus2 = self.soup.find_all('li') #obtain all paragraphs starting with tag <li>\n",
    "\n",
    "        #Get paragraphs from wordcorpus with special format into a list\n",
    "        self.para=[]\n",
    "        for paragraph in self.wordcorpus: #append paragraphs starting with <p>\n",
    "            self.para.append(paragraph)\n",
    "\n",
    "        self.relatedtopic = \",*RELATED WIKI TOPIC*\" #Identify topics in Wikipedia with an URL to point out to user\n",
    "        for paragraph in self.wordcorpus2: #append paragraphs starting with <li>\n",
    "            if str(paragraph).find('<li><a href=') != -1:\n",
    "                if str(paragraph).find('</a></li>') != -1 or str(paragraph).find('</a></sup></li>') != -1: \n",
    "                    self.para.append(self.relatedtopic)\n",
    "            if str(paragraph).find('toctext') == -1: #remove Wiki headers 1.2.3 with toctext as they can't be arranged properly\n",
    "                self.para.append(paragraph)\n",
    "    \n",
    "        #REASON WHY WE HAVE TO DO TWO FOR LOOPS WITH TWO wordcorpus IS BECAUSE THE FIND_ALL FOR ARRAY IS NOT IN ORDER\n",
    "        #COMMENCE CLEANING OF UNWANTED HTML <> and WIKI LINK [no]\n",
    "        \n",
    "        #For FIXING the summary function\n",
    "        self.troubleshoot = self.para\n",
    "        \n",
    "        #DATA CLEANING OF UNWANTED HTML <> and []\n",
    "        self.para = list(str(self.para)) #chop everything into letters for cleaning\n",
    "        \n",
    "        #This block of code removes the first letter [, removes any words with <> html tag or [] citation\n",
    "        #When it detects a <li> it will create two blanks\n",
    "        self.start = 0 #is letter currently inside tag <>\n",
    "        self.end = 0 #has <> just ended, need to check for , if it just ended to not copy a comma after <>\n",
    "        self.first = 1 #first letter is [, need to omit\n",
    "        self.bracket = 0 #check if letter is inside bracket\n",
    "        self.li = 0 #check for <li> to line break\n",
    "        self.p = 0 #check for <p> to line break\n",
    "        self.point = 0 #after <li> or list, puts a • before adding new letter\n",
    "        self.para2 = []\n",
    "        for letter in self.para:\n",
    "            if self.first == 0:\n",
    "                if letter == '<': #tells python to stop reading letters inside a bracket\n",
    "                    self.start = 1\n",
    "                elif letter == '>': #next letter can be read since its out of bracket, unless its another <\n",
    "                    self.start = 0\n",
    "                    self.end = 1\n",
    "                elif self.end == 1 and letter == ',': #skip COMMA reading when it occurs like </p>, at end of para\n",
    "                    self.end = 0\n",
    "                    continue\n",
    "                elif letter == '[':\n",
    "                    self.bracket = 1\n",
    "                    self.end = 0\n",
    "                elif letter == ']':\n",
    "                    self.bracket = 0\n",
    "                    self.end = 0\n",
    "                elif self.start == 0 and self.bracket == 0: #ALL CLEAR TO READ LETTER\n",
    "                    self.end = 0\n",
    "                    if self.point == 1:\n",
    "                        self.para2.append('• ')\n",
    "                        self.point = 0\n",
    "                    self.para2.append(letter)\n",
    "            if letter == '<':\n",
    "                self.li = 1\n",
    "            elif letter != 'l' and self.li == 1:\n",
    "                self.li = 0\n",
    "            elif letter == 'l' and self.li == 1:\n",
    "                self.li = 2\n",
    "            elif letter == 'i' and self.li == 2:\n",
    "                self.li = 3\n",
    "            elif letter != '>' and self.li == 3:\n",
    "                self.li = 0\n",
    "            elif letter == '>' and self.li == 3:\n",
    "                self.para2.append('\\n\\n')\n",
    "                self.li = 0\n",
    "                self.point = 1\n",
    "            if letter == '<':\n",
    "                self.p = 1\n",
    "            elif letter != 'p' and self.p == 1:\n",
    "                self.p = 0\n",
    "            elif letter == 'p' and self.p == 1:\n",
    "                self.p = 2\n",
    "            elif letter == '>' and self.p == 2:\n",
    "                self.para2.append('\\n')\n",
    "                self.p = 0\n",
    "            self.first = 0 #Had an issue with the first letter being [, after skipping this, the [number] checks can run\n",
    "\n",
    "        self.para2 =''.join(self.para2) #combine back all letters and spaces\n",
    "        \n",
    "        #REMOVE UNWANTED ARRAYS\n",
    "        self.para = []\n",
    "        \n",
    "        #WORD COUNT (SELF.PARA3) AND COMMON WORDS (SELF.WORDCOUNTER)\n",
    "        \n",
    "        self.para3 = self.para2.split() #split paragraphs into words again for counting\n",
    "        self.niceword = ''\n",
    "        self.punctuation = ('.',',','(',')','\"',\"'\",'?','!','*','|',':',';') \n",
    "        \n",
    "        if self.to_lemmatize == True:\n",
    "            for index, word in enumerate(self.para3):\n",
    "                self.niceword = word\n",
    "                self.niceword = self.niceword.lower() #standardize all to lower case before counting\n",
    "                for punctuation in self.punctuation:\n",
    "                    self.niceword = self.niceword.replace(punctuation,'') #clean up bad punctuation\n",
    "                # Lemmatize word if configured in wiki object\n",
    "                self.niceword = self.lemmatizer.lemmatize(self.niceword)\n",
    "                self.para3[index] = self.niceword\n",
    "        else:\n",
    "            for index, word in enumerate(self.para3):\n",
    "                self.niceword = word\n",
    "                self.niceword = self.niceword.lower() #standardize all to lower case before counting\n",
    "                for punctuation in self.punctuation:\n",
    "                    self.niceword = self.niceword.replace(punctuation,'') #clean up bad punctuation\n",
    "                self.para3[index] = self.niceword\n",
    "        \n",
    "        \n",
    "        self.wordcounter = Counter(self.para3) \n",
    "        #Counter solely used for word count, will use banlist to remove words from it later. \n",
    "        #Make new wordcounter2 + banlist for use\n",
    "        self.allowedwords = dict(self.wordcounter.most_common()) \n",
    "        #convert to dictionary so that for loop can extract words + do unique word count + total word count\n",
    "        \n",
    "\n",
    "        #FIND OUT UNIQUE WORD COUNT AND TOTAL WORD COUNT BEFORE BANLIST - SELF.FULLWORDS AND SELF.FULLCOUNT\n",
    "        self.fullcount = 0\n",
    "        self.fullwords = 0\n",
    "        for key in self.allowedwords:\n",
    "            self.fullcount += self.allowedwords[key]\n",
    "            self.fullwords += 1\n",
    "        \n",
    "        #IMPLEMENT BAN LIST (FROM WIKIPEDIA) BY DEL FUNCTION FOR COUNTER SELF.WORDCOUNTER AND WORD LIST SELF.ALLOWEDWORDS\n",
    "        banlist = ('the', 'be', 'to', 'of', 'and', 'a', 'in', 'that', 'have', 'I', 'it', 'for', 'not', 'on', 'with', \n",
    "                   'he', 'as', 'you', 'do', 'at', 'this', 'but', 'his', 'by', 'from', 'they', 'we', 'say', 'her', 'she', \n",
    "                   'or', 'an', 'will', 'my', 'one', 'all', 'would', 'there', 'their', 'what', 'so', 'up', 'out', 'if', \n",
    "                   'about', 'who', 'get', 'which', 'go', 'me', 'when', 'make', 'can', 'like', 'time', 'no', 'just', \n",
    "                   'him', 'know', 'take', 'people', 'into', 'year', 'your', 'good', 'some', 'could', 'them', 'see', \n",
    "                   'other', 'than', 'then', 'now', 'look', 'only', 'come', 'its', 'over', 'think', 'also', 'back', \n",
    "                   'after', 'use', 'two', 'how', 'our', 'work', 'first', 'well', 'way', 'even', 'new', 'want','topic', \n",
    "                   'because', 'any', 'these', 'give', 'day', 'most', 'us','retrieved','^','archived',\"•\",'related',\n",
    "                   \"',*related\",\"wiki\",\"topic*',\",\"is\",\"are\",'was','since','such','articles','has','&amp;','p','b',\n",
    "                  'january','february','march','april','may','june','july','august','september','october','november',\n",
    "                  'december','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17','18','19','20',\n",
    "                  '21','22','23','24','25','26','27','28','29','30','31','i','wa')\n",
    "        \n",
    "        #English banlist from top 100 common words and some extra terms\n",
    "        if self.lang == 'en':\n",
    "            for word in banlist: #delete words in counter and list only if it's in english\n",
    "                del self.wordcounter[word]\n",
    "                self.allowedwords.pop(word, None)\n",
    "            \n",
    "        #If NLTK stoplist is used, delete words from their stoplist as well\n",
    "        if self.nltkrun == True:\n",
    "            for word in self.nltkstopword: #delete words in counter and list\n",
    "                del self.wordcounter[word]\n",
    "                self.allowedwords.pop(word, None)\n",
    "            \n",
    "        #DELETE '',.,·,•,↑,space,null,related,wiki,common from counter and dictionary\n",
    "        banlist2 = (\"''\",\".\",\"·\",\"•\",\"↑\",\" \",\"\",\"-\",\"–\",\"/\",\"related\",\"wiki\",\"common\")\n",
    "        for char in banlist2:\n",
    "            del self.wordcounter[char]\n",
    "            self.allowedwords.pop(char, None)\n",
    "        \n",
    "        #FIND OUT TOTAL WORD COUNT AFTER BANLIST FOR FUNCTION COMMONWORDPCT THAT USES PERCENTAGE OF THRESHOLD FOR WORD COUNT\n",
    "        #SELF.FULLCOUNT2 IS TOTAL WORDS AFTER BANLIST AND FULLWORDS2 IS UNIQUE WORD COUNT AFTER BANLIST\n",
    "        self.fullcount2 = 0\n",
    "        self.fullwords2 = 0\n",
    "        \n",
    "        # SELF.ALLOWEDWORDS contain all the words and frequency after banlist, used for functions listed below\n",
    "        for key in self.allowedwords: #self.allowedwords HAVE WORDS REMOVED ON ITSELF VIA BANLIST, DONT NEED NEW VARIABLE\n",
    "            self.fullcount2 += self.allowedwords[key] #FULLCOUNT2 used for COMMONWORDPCT and TOTAL WORDS function to show total word count aft removing banlist\n",
    "            self.fullwords2 += 1 #FULLWORDS2 used in TOTAL WORDS function to show unique word count aft removing banlist\n",
    "        \n",
    "    \n",
    "        #This section checks if the Wiki site was loaded successfully..\n",
    "        self.missing = self.soup.find_all('b') \n",
    "        #Wikipedia does not have an article with this exact name.\n",
    "        #This sentence that always appears for Error 404 pages, is bolded, so <b> tag can help to find it\n",
    "                \n",
    "        #Check for sentence that tells of Error 404 website using a counter.\n",
    "        self.goodsite = 1\n",
    "        self.offsite = 0\n",
    "        \n",
    "        #Check if a site goes through but it is an ambiguous site (recommendations page)\n",
    "        for sentence in self.wordcorpus: \n",
    "            #refer to: phrase belongs in a <p> paragraph\n",
    "            if str(sentence).find(\"refer to:\") != -1:\n",
    "                self.offsite = 1            \n",
    "        \n",
    "        #Check if Wikipedia article exists\n",
    "        for sentence in self.missing: \n",
    "            if str(sentence) == \"<b>Wikipedia does not have an article with this exact name.</b>\": #CONVERT ELEMENT TO STRING TYPE BEFORE CHECK!!!\n",
    "                self.goodsite = 0 #sentence exists, bad site means counter flips to 0\n",
    "        \n",
    "        #Confirmation for successful Wikipedia site load\n",
    "        if self.goodsite == 1 and self.offsite == 0:\n",
    "            print('Wikipedia page loaded successfully!! Type variablename.HELP() for documentation of functions.')\n",
    "        \n",
    "            #WHEN SEARCH IS SUCCESSFUL, PRINT 2 PARAGRAPHS FOR PREVIEW\n",
    "            self.parashort = []\n",
    "            self.noofpara = 0\n",
    "            for paragraph in self.wordcorpus: #append ONLY 2 paragraphs starting with <p>\n",
    "                if self.noofpara < 2 and str(paragraph) != '<p class=\"mw-empty-elt\">\\n</p>' and len(str(paragraph)) > 199: \n",
    "                    self.parashort.append(paragraph)\n",
    "                    self.noofpara += 1\n",
    "\n",
    "            #Data cleaning for printing out summary of Wikipedia (2 paragraphs) if search is successful - __cleantext_summary\n",
    "            self.parashort2 = self.__cleantext_summary(self.parashort)\n",
    "            \n",
    "            #REMOVE UNWANTED ARRAYS\n",
    "            self.parashort = []        \n",
    "\n",
    "            #END OF DATA PROCESSING FOR AUTOMATIC SUMMARY PRINTED WHEN SEARCHING\n",
    "\n",
    "            #Print two paragraphs when search is successful\n",
    "            print(self.parashort2)\n",
    "            \n",
    "            \n",
    "        #Ambiguous search detecton and suggested article names\n",
    "        elif self.goodsite == 1 and self.offsite == 1:\n",
    "            print('\\nThe title \"'+ self.title.replace(\"_\", \" \") + '\" you specified is ambiguous. As a result, you are linked to a clarification page.\\n\\n')\n",
    "            print('Here are some suggestions to use: \\n')\n",
    "            \n",
    "            self.all_links = self.soup.find_all(\"a\") #ALL HTML TAGS STARTING WITH <A, E.G. <A HREF, <A TITLE AND FULL PARAGRAPH\n",
    "            self.wiktwords = []\n",
    "            for link in self.all_links:\n",
    "                self.wiktwords.append(link.get(\"title\")) #TAG STARTING WITH A, CONTENT ENCLOSED INSIDE TITLE=\"\"\n",
    "                #print(link.get(\"title\")) #shows list of items appended, common words all start with wikt:\n",
    "\n",
    "            self.cleanlink = []\n",
    "            for words in self.wiktwords: \n",
    "                self.words2 = str(words) #words are not string yet so need str function before saving into new var\n",
    "                self.cleanlink.append(self.words2)\n",
    "\n",
    "            for link in self.cleanlink:\n",
    "                if link.find(\"Help:\") != -1:\n",
    "                    break\n",
    "                elif link.find(\"Edit section:\") != -1:\n",
    "                    continue\n",
    "                else:\n",
    "                    print(link)\n",
    "            \n",
    "        else:\n",
    "            #Unable to fnd Wikipedia Page\n",
    "            print('Wikipedia page could not be found for \"' + str(self.title.replace(\"_\", \" \")) + '\". Please try again!') \n",
    "            print('Other useful information: Enclose title argument with single quotes. Spaces are allowed, and title is case insensitive.')\n",
    "        \n",
    "        \n",
    "    def __cleantext_summary(self, corpus):\n",
    "        '''Gets summary of the text, internal method'''\n",
    "        #Data cleaning for printing out summary of Wikipedia (2 paragraphs) if search is successful\n",
    "        corpus = list(str(corpus)) #chop everything into letters for usage\n",
    "        start = 0 \n",
    "        end = 0 \n",
    "        first = 1 \n",
    "        bracket = 0 \n",
    "        li = 0 \n",
    "        p = 0 \n",
    "        corpus2 = []\n",
    "        for letter in corpus:\n",
    "            if first == 0:\n",
    "                if letter == '<': #tells python to stop reading letters inside a bracket\n",
    "                    start = 1\n",
    "                elif letter == '>': #next letter can be read since its out of bracket, unless its another <\n",
    "                    start = 0\n",
    "                    end = 1\n",
    "                elif end == 1 and letter == ',': #skip COMMA reading when it occurs like </p>, at end of para\n",
    "                    end = 0\n",
    "                    continue\n",
    "                elif letter == '[':\n",
    "                    bracket = 1\n",
    "                    end = 0\n",
    "                elif letter == ']':\n",
    "                    bracket = 0\n",
    "                    end = 0\n",
    "                elif start == 0 and bracket == 0: #ALL CLEAR TO READ LETTER\n",
    "                    end = 0\n",
    "                    corpus2.append(letter)\n",
    "            if letter == '<':\n",
    "                p = 1\n",
    "            elif letter != 'p' and p == 1:\n",
    "                p = 0\n",
    "            elif letter == 'p' and p == 1:\n",
    "                p = 2\n",
    "            elif letter == '>' and p == 2:\n",
    "                corpus2.append('\\n\\n')\n",
    "                p = 0\n",
    "            first = 0 #Had an issue with the first letter being [, after skipping this, the [number] checks can run\n",
    "\n",
    "        corpus2 = ''.join(corpus2) #combine back all letters and spaces\n",
    "        return corpus2\n",
    "    \n",
    "    #Retrieves full text of Wikipedia Article, 'Yes' for 2nd argument to output/return it instead of print\n",
    "    def gettext(self,outfull='No'): #comes after init\n",
    "        '''gettext accepts 1 optional argument - Yes to output string and No to print text (default: No). It retrieves the full text of the Wikipedia title.'''\n",
    "        if isinstance(outfull, str): #check for string yes, no and other permutations\n",
    "            if outfull.lower().strip() in {'yes','true','y','t'}:\n",
    "                print(\"Full text is output. To print it instead, put 'no' in argument\")\n",
    "                return self.para2\n",
    "            elif outfull.lower().strip() in {'no','false','n','f','na','n/a','nan'}:\n",
    "                print(\"Full text is printed. To set as output, put 'yes' in argument\")\n",
    "                print(self.para2)\n",
    "            else:\n",
    "                print(\"Full text is invalid, summary is printed by default. To set as output, put 'yes' in argument\")\n",
    "                print(self.para2)\n",
    "        elif isinstance(outfull, bool): #check for boolean yes/no\n",
    "            if outfull == True:\n",
    "                print(\"Full text is output. To print it instead, put False or 'no' in argument\")\n",
    "                return self.para2\n",
    "            else:\n",
    "                print(\"Full text is printed. To set as output, put True or 'yes' in argument\")\n",
    "                print(self.para2)   \n",
    "        else: #run default - print if invalid option\n",
    "            print(\"2nd argument not a string, full text is printed by default. To set as output, put 'yes' in argument\")\n",
    "            print(self.para2)\n",
    "\n",
    "    #Show frequency count of most common words, able to select number of words to output\n",
    "    def commonwords(self,wordcount=100):\n",
    "        '''commonwords accepts 1 optional argument (default: 100) for the number of most common words in the site and their frequencies to show.'''\n",
    "        self.wordcount = 100\n",
    "        if wordcount != 100 and isinstance(wordcount, int) == True:\n",
    "            self.wordcount = wordcount\n",
    "        elif wordcount != 100 and isinstance(wordcount, int) == False:\n",
    "            print('Word count specified is currently not an integer. Hence default of 100 words is used\\n')\n",
    "        #convert counter to list to dictionary then sum up total word count using for loop in word[key]\n",
    "        self.topwords = dict(self.wordcounter.most_common(self.wordcount))\n",
    "        return self.topwords\n",
    "        \n",
    "    #Show frequency count of most common words based on percentage of word count in the article\n",
    "    #e.g. 10% of 10,000 words is 1,000 words, the N most common words with total frequency =< 1000 will be retrieved\n",
    "    def commonwordspct(self,percent=10):\n",
    "        '''commonwordspct accepts 1 optional argument (default: 10) on the percentage threshold of word count to determine the most frequent words to show.'''\n",
    "        self.percent = 10\n",
    "        if isinstance(percent, int) == True or isinstance(percent, float) == True:\n",
    "            if percent != 10 and 0 < percent <= 100:\n",
    "                self.percent = percent\n",
    "        else:\n",
    "            print('The percent specified is either not an integer or float, 0% or over 100% which is not allowed. Hence, 10% set by default.\\n')\n",
    "        \n",
    "        self.wordcount2 = int(self.percent/100*self.fullcount2) #full count 2 is word count after banlist\n",
    "        print(str(self.percent) + \"% threshold means the most common words shown/output below appeared equal to or less than \" + str(self.wordcount2) + ' times out of ' + str(self.fullcount2) + \" total word count.\\n\")\n",
    "        self.partialcount = 0\n",
    "        self.partialwordlist = []\n",
    "        for key in self.wordcounter:\n",
    "            self.partialcount += self.wordcounter[key]\n",
    "            if self.partialcount <= self.wordcount2:\n",
    "                self.partialwordlist.append(key)\n",
    "            else:\n",
    "                break\n",
    "        self.partialcounttowordno = 0\n",
    "        for word in self.partialwordlist:\n",
    "            self.partialcounttowordno += 1\n",
    "        self.topwords2 = dict(self.wordcounter.most_common(self.partialcounttowordno))\n",
    "        if self.partialcounttowordno != 0:\n",
    "            print('For the word count threshold, the ' + str(self.partialcounttowordno) + ' most common words are shown/output.\\n\\n')\n",
    "            return self.topwords2\n",
    "        else:\n",
    "            print('The most common word has a percentage occurance higher than the threshold set as the percentage set is too low.\\n')\n",
    "    \n",
    "    #Determine total word count and unique word count after banlist/word stoplist\n",
    "    def totalwords(self): #word count are all BEFORE banlist\n",
    "        '''totalwords accepts 0 argument and shows the total word count and unique word count'''\n",
    "        print('Total word count is ' + str(self.fullcount))\n",
    "        print('Total word count is ' + str(self.fullcount2) + ' after implementing banlist\\n')\n",
    "        print('Unique word count is ' + str(self.fullwords) + ' for the Wikipedia site titled ' + str(self.title.replace(\"_\", \" \")))\n",
    "        print('Unique word count AFTER BANLIST is ' + str(self.fullwords2) + ' for the Wikipedia site titled ' + str(self.title.replace(\"_\", \" \")))\n",
    "        return [self.fullcount,self.fullcount2,self.fullwords,self.fullwords2]\n",
    "    \n",
    "    #Plot the most common words, 2nd argument allows you to choose number of words to plot, and 3rd arg is the Nth most common word to start plotting from\n",
    "    def plotwords(self,graphname='wordcount',wordcount2=20,startword=1,removeyear=10):\n",
    "        '''plotwords accepts 4 optional arguments. \n",
    "        The first argument is the filename to save as (default: wordcount.png).\n",
    "        The second argument (default: 20) is for the number of most frequent words to show as a GRAPH. \n",
    "        The third argument is the Nth most frequent word to start plotting from. (default: 1, starting from most frequent word).\n",
    "        The fourth argument removes the latest N years from the most frequent words (default: remove latest 10 years)'''\n",
    "        if isinstance(wordcount2, int) == True and isinstance(startword, int) == True:\n",
    "            if startword < 1 or wordcount2 < 1:\n",
    "                self.notify = 2 #Error as out of range, use default\n",
    "                self.wordcount2 = 20\n",
    "                self.startword = 1\n",
    "            else:\n",
    "                self.wordcount2 = wordcount2\n",
    "                self.startword = startword\n",
    "                self.notify = 0\n",
    "        else:\n",
    "            self.notify = 1 #Error as not integer input, use default\n",
    "            self.wordcount2 = 20\n",
    "            self.startword = 1\n",
    "            \n",
    "        if self.notify == 1:\n",
    "            print('Word count or start position specified is currently not an integer. Hence default of 20 words starting from 1st word is used for graph\\n')\n",
    "        elif self.notify == 2:\n",
    "            print('Word count or start position specified must be 1 or greater. Default of 20 words starting from 1st word is used for graph\\n')\n",
    "            \n",
    "        # Change file name\n",
    "        if isinstance(graphname, str) == True:\n",
    "            self.graphname = graphname + '.png'\n",
    "        else:\n",
    "            self.graphname = 'wordcount.png'\n",
    "\n",
    "        #convert counter to list to dictionary then sum up total word count using for loop in word[key]\n",
    "        #if start position is not modified (start from most common word, use default dict)\n",
    "        #otherwise, have to make a new dictionary for plotting graph by getting start th to start + wordcount th words\n",
    "        \n",
    "        self.curyear = datetime.datetime.now().year\n",
    "        # Banlist, omit the last n years in plotwords graph\n",
    "        self.yearban = ['0000'] \n",
    "        \n",
    "        if isinstance(removeyear, int) == True:\n",
    "            if removeyear >= 0:\n",
    "                self.removeyear = removeyear\n",
    "            else:\n",
    "                self.removeyear = 10 # Error as years to remove cannot be negative\n",
    "                print('Number of latest years to exclude in word frequency graph cannot be negative. Excluding the most recent 10 years by default, starting from ' + str(self.curyear))\n",
    "        else:\n",
    "            self.removeyear = 10 # Error as not integer input, use default\n",
    "            print('Number of latest years to exclude in word frequency graph is invalid. Excluding the most recent 10 years by default, starting from ' + str(self.curyear))\n",
    "\n",
    "        for i in range(self.removeyear):\n",
    "            self.yearban.append(str(self.curyear - i))\n",
    "            if self.curyear - i == 0:\n",
    "                break\n",
    "        # print(self.yearban)\n",
    "        \n",
    "        # Store words and freq in dictionary\n",
    "        self.topwords2 = {}\n",
    "        self.wordno_graph = 0\n",
    "        for i, (word, freq) in enumerate(dict(self.wordcounter.most_common()).items()):\n",
    "            # Since index i starts first word with i=0 and startword min is 1, need to check i+1 with startword min\n",
    "            if i+1 >= self.startword and self.wordno_graph < self.wordcount2 and word not in self.yearban:\n",
    "                #Add to dictionary if its between x - x+i words in the dictionary and not a year\n",
    "                self.topwords2[word] = freq\n",
    "                self.wordno_graph += 1\n",
    "            elif self.wordno_graph == self.wordcount2:\n",
    "                break\n",
    "                #stop loop once required words are retrieved\n",
    "              \n",
    "                    \n",
    "        #matplotlib.pyplot.bar(range, height, tick_label)\n",
    "        self.wordnames = list(self.topwords2.keys())\n",
    "        self.wordvalues = list(self.topwords2.values())\n",
    "        self.wordnames.reverse()\n",
    "        self.wordvalues.reverse()\n",
    "\n",
    "        #tick_label does the some work as plt.xticks()\n",
    "        plt.figure(figsize=(22, 18), dpi= 80, facecolor='w', edgecolor='k')\n",
    "        plt.rc('ytick', labelsize=math.ceil(30*math.exp(-self.wordno_graph*0.05))+7) \n",
    "        plt.rc('xtick', labelsize=20) \n",
    "        plt.style.use('ggplot')\n",
    "        self.localgraph = plt.barh(range(len(self.topwords2)),self.wordvalues,tick_label=self.wordnames)\n",
    "        plt.title('Word Frequency of Wiki Article: ' + self.graphtitle + ' for the Top ' + str(self.wordno_graph) + ' words, starting from word number ' + str(self.startword),fontsize=22)\n",
    "        \n",
    "        #Colored bar graphs divided by green (most frequent words), orange (moderate), red (not as frequent)\n",
    "        for i in range(self.wordno_graph):\n",
    "            if i <= float(self.wordno_graph)/3:\n",
    "                self.localgraph[i].set_color('red')\n",
    "            elif i <= float(2*self.wordno_graph)/3:\n",
    "                self.localgraph[i].set_color('orange')\n",
    "            else:\n",
    "                self.localgraph[i].set_color('green')\n",
    "        \n",
    "        plt.savefig(self.graphname)\n",
    "        plt.rcParams['figure.figsize'] = [22, 18]\n",
    "        plt.show()\n",
    "                \n",
    "        self.cwd = os.getcwd()\n",
    "        print('Graph is saved as ' + self.graphname + ' in directory: ' + str(self.cwd))\n",
    "    \n",
    "\n",
    "\n",
    "    #Plot the most commonly cited Years in the article for time analytics, able to change number of Years to show\n",
    "    def plotyear(self,graphname='yearcount',yearcount3=20):\n",
    "        '''plotyear accepts 2 optional arguments. The first argument is the filename to save as (default: yearcount.png).\n",
    "        The second argument (default: 20) is the number of years to plot in the graph. \n",
    "        The frequency count of the most common years will be plotted. \n",
    "        This allows the user to understand the years of interest for the Wikipedia Topic.'''\n",
    "        if isinstance(yearcount3, int) == True:\n",
    "            if yearcount3 < 1:\n",
    "                self.notify = 2 #Error as out of range, use default\n",
    "                self.yearcount3 = 20\n",
    "            else:\n",
    "                self.yearcount3 = yearcount3\n",
    "                self.notify = 0 #OK\n",
    "        else:\n",
    "            self.notify = 1 #Error as not integer input, use default\n",
    "            self.yearcount3 = 20\n",
    "\n",
    "        # Change file name\n",
    "        if isinstance(graphname, str) == True:\n",
    "            self.graphname = graphname + '.png'\n",
    "        else:\n",
    "            self.graphname = 'yearcount.png'\n",
    "            \n",
    "        self.tempfulllist = dict(self.wordcounter.most_common())\n",
    "        self.yearlist = {}\n",
    "        self.actualyearcount = 0 #counter for number of years              \n",
    "        \n",
    "        for year in self.tempfulllist:\n",
    "            if re.search(r\"\\b\\d{4}\\b\", year) != None and year.find('-') == -1 and year.find('–') == -1 and year[0:2] != '00':\n",
    "                try:\n",
    "                    self.yearlist[year] = self.tempfulllist[year]\n",
    "                    self.actualyearcount += 1\n",
    "                    if self.actualyearcount == self.yearcount3:\n",
    "                        break\n",
    "                except:\n",
    "                    continue  \n",
    "        \n",
    "        #matplotlib.pyplot.bar(range, height, tick_label)\n",
    "        self.yearnames = list(self.yearlist.keys())\n",
    "        self.yearvalues = list(self.yearlist.values())\n",
    "        self.yearnames.reverse()\n",
    "        self.yearvalues.reverse()\n",
    "\n",
    "        #tick_label does the some work as plt.xticks()\n",
    "        plt.figure(figsize=(22, 18), dpi= 80, facecolor='w', edgecolor='k')\n",
    "        plt.rc('ytick', labelsize=math.ceil(53*math.exp(-self.actualyearcount*0.06))+7) \n",
    "        plt.rc('xtick', labelsize=20) \n",
    "        plt.style.use('ggplot')\n",
    "        self.yeargraph = plt.barh(range(len(self.yearlist)),self.yearvalues,tick_label=self.yearnames)\n",
    "        plt.title('Interest in ' + self.graphtitle + ' over the years measured by Frequency Count of each Year',fontsize=22)\n",
    "        \n",
    "        for i in range(self.actualyearcount):\n",
    "            if i <= float(self.actualyearcount)/3:\n",
    "                self.yeargraph[i].set_color('red')\n",
    "            elif i <= float(2*self.actualyearcount)/3:\n",
    "                self.yeargraph[i].set_color('orange')\n",
    "            else:\n",
    "                self.yeargraph[i].set_color('green')\n",
    "    \n",
    "        plt.savefig(self.graphname)\n",
    "        plt.rcParams['figure.figsize'] = [22, 18]\n",
    "        plt.show()    \n",
    "        \n",
    "        if self.notify == 1:\n",
    "            print('Year count specified is currently not an integer. Hence default of 20 years is used for graph\\n')\n",
    "        elif self.notify == 2:\n",
    "            print('Year count specified must be 1 or greater. Default of 20 years is used for graph\\n')\n",
    "            \n",
    "        self.cwd = os.getcwd()\n",
    "        print('Graph is saved as ' + self.graphname + ' in directory: ' + str(self.cwd))\n",
    "    \n",
    "    #Retrieve summary of Wikipedia article, able to choose number of paragraphs (1st arg), and to return/output as string (2nd arg) instead of print\n",
    "    def summary(self, paravalue=2, outsummary='no'):\n",
    "        '''summary accepts 2 optional arguments, the first one for the number of paragraphs to show (default: 2) and the second one - Yes to output string and No to print text (default: No). It gives a summary of the Wikipedia page.'''\n",
    "        self.paravalue = 2\n",
    "        if isinstance(paravalue, int) == True:\n",
    "            if paravalue > 0:\n",
    "                self.paravalue = paravalue\n",
    "            else:\n",
    "                print('The number of paragraphs specified is not valid. Default 2 paragraphs will be displayed \\n')\n",
    "        else:\n",
    "            print('The number of paragraphs specified is not an integer. Default 2 paragraphs will be displayed \\n')\n",
    "        \n",
    "        #MAKE AN ARRAY AND STRING FOR JUST N PARAGRAPHS!\n",
    "        self.parashort = []\n",
    "        self.noofpara = 0\n",
    "        for paragraph in self.wordcorpus: #append ONLY 2 paragraphs starting with <p>\n",
    "            if self.noofpara < self.paravalue and str(paragraph) != '<p class=\"mw-empty-elt\">\\n</p>' and len(str(paragraph)) > 199: \n",
    "                self.parashort.append(paragraph)\n",
    "                self.noofpara += 1\n",
    "\n",
    "        self.parashort2 = self.__cleantext_summary(self.parashort)\n",
    "        #REMOVE UNWANTED ARRAYS\n",
    "        self.parashort = []    \n",
    "        \n",
    "        #END OF DATA PROCESSING FOR SUMMARY FUNCTION! i.e. 2 paragraph of wiki\n",
    "\n",
    "        #Checking if you should print or output argument\n",
    "        if isinstance(outsummary, str): #check for string yes, no and other permutations\n",
    "            if outsummary.lower().strip() in {'yes','true','y','t'}:\n",
    "                print(\"Summary is output. To print it instead, put 'no' in argument\")\n",
    "                return self.parashort2\n",
    "            elif outsummary.lower().strip() in {'no','false','n','f','na','n/a','nan'}:\n",
    "                print(\"Summary is printed. To set as output, put 'yes' in 2nd argument\")\n",
    "                print(self.parashort2)\n",
    "            else:\n",
    "                print(\"2nd argument is invalid, summary is printed by default. To set as output, put 'yes' in 2nd argument\")\n",
    "                print(self.parashort2)\n",
    "        elif isinstance(outsummary, bool): #check for boolean yes/no\n",
    "            if outsummary == True:\n",
    "                print(\"Summary is output. To print it instead, put False or 'no' in 2nd argument\")\n",
    "                return self.parashort2\n",
    "            else:\n",
    "                print(\"Summary is printed. To set as output, put True or 'yes' in 2nd argument\")\n",
    "                print(self.parashort2)   \n",
    "        else: #run default - print if invalid option\n",
    "            print(\"2nd argument not a string, summary is printed by default. To set as output, put 'yes' in 2nd argument\")\n",
    "            print(self.parashort2)\n",
    "            \n",
    "    #HELP and basic documentation of available functions\n",
    "    def HELP(self):\n",
    "        '''Explains how to use the class object wiki and also retrieves a list of methods with their actions.'''\n",
    "        print('The wiki() class accepts 5 arguments. The first one is a compulsory title of the Wikipedia page. Second is to format the search string to proper/title case (Yes/No, default: Yes).')\n",
    "        print('Third is for language settings (e.g. English, de, francais, etc., default: English).')\n",
    "        print('Fourth is for implementing NLTK stoplist in provided language based on 3rd arg (Yes/No, default: standard stoplist provided).')\n",
    "        print('Fifth is for lemmatizing text (Yes/No, default: No).\\n\\n')\n",
    "        print('Functions/Methods of Wikipedia scraper package: \\n')\n",
    "        print('commonwords accepts 1 optional argument (default: 100) for the number of most common words in the site and their frequencies to show.\\n')\n",
    "        print('commonwordspct accepts 1 optional argument (default: 10) on the percentage threshold of word count to determine the most frequent words to show.\\n')\n",
    "        print('plotwords accepts 4 optional arguments. The first argument is the filename to save as (default: wordcount.png). The second argument (default: 20) is for the number of most frequent words to show as a GRAPH. The third argument is the Nth most frequent word to start plotting from. (default: 1, starting from most frequent word). The third argument is the filename to save as. The fourth argument removes the latest N years from the most frequent words (default: remove latest 10 years)\\n')\n",
    "        print('plotyear accepts 2 optional argument. The first argument is the filename to save as (default: yearcount.png). The second argument (default: 20) is the number of years to plot in the graph. The frequency count of the most common years will be plotted. This allows the user to understand the years of interest for the Wikipedia Topic.\\n')\n",
    "        print('totalwords accepts 0 argument and shows the total word count and unique word count\\n')\n",
    "        print('summary accepts 2 optional arguments, the first one for the number of paragraphs to show (default: 2) and the second one - Yes to output string and No to print text (default: No). It gives a summary of the Wikipedia page\\n')\n",
    "        print('gettext accepts 1 optional argument - Yes to output string and No to print text (default: No). It retrieves the full text of the Wikipedia title\\n')\n",
    "        "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}