SBD-rulebased-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e148d974",
   "metadata": {},
   "source": [
    "### Sentance Boundary Disambiguation - Rule Based Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dd9a4a3",
   "metadata": {},
   "source": [
    "* 1) Develop a rule based or ML based sentence boundary disambiguation system for the language of your choice.\n",
    "* 2) Write observations on the accuracy of your model and also elucidate any methods you can suggest to improve the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c26c75d",
   "metadata": {},
   "source": [
    "Sentence boundary disambiguation (SBD), also known as sentence breaking, sentence boundary detection, and sentence segmentation, is the problem in natural language processing of deciding where sentences begin and end. Natural language processing tools often require their input to be divided into sentences; however, sentence boundary identification can be challenging due to the **potential ambiguity of punctuation marks**. In written English, a period may indicate the end of a sentence, or may denote an **abbreviation, a decimal point, an ellipsis, or an email address**, among other possibilities.\n",
    "\n",
    "#### References\n",
    "* https://grammar.yourdictionary.com/punctuation/what/fourteen-punctuation-marks.html\n",
    "* https://aclanthology.org/J97-2002.pdf\n",
    "* https://dbpedia.org/page/Sentence_boundary_disambiguation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdfeacf0",
   "metadata": {},
   "source": [
    "### Steps followed:\n",
    "* 1. First split sentances into words based on space.\n",
    "* 2. Then based on the terminating punctuations (.?!) split the sentances into **leftword + punctuation + rightword** and add ending marker at the potential boundary places\n",
    "* 3. Then past the split sentances through rules and check if the boundaries are correct. Store the correct boundaries in a \n",
    "     boundary list\n",
    "* 4. Depending on the final boundary list obtained add boundary marker\n",
    "* 5. Split the string based on the boundary marker\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2d781c1",
   "metadata": {},
   "source": [
    "## Helper Functions\n",
    "\n",
    "Making helper functions for checking certain conditions like:\n",
    "* terminating punctuations\n",
    "* numeric character\n",
    "* double quotes\n",
    "* abbreviation type 1\n",
    "* abbreviation type 2\n",
    "* word is capital \n",
    "* word starting and ending with double quotes\n",
    "* checking url\n",
    "* checking email\n",
    "* check elipsis\n",
    "* check decimals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c7d4051d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# terminating punctuations\n",
    "\n",
    "def isTermPunct(character):\n",
    "    punct = [\".\",\"!\",\"?\"]\n",
    "    flag=0\n",
    "    for i in punct:\n",
    "#         print(punct[i]) \n",
    "       if(i==character):\n",
    "            flag=1\n",
    "\n",
    "    if(flag==1):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "    \n",
    "    \n",
    "# numbers\n",
    "\n",
    "def checkNumChar(character):\n",
    "    numchar = ['#','$','%','+','>','<','/']\n",
    "    flag=0\n",
    "    for i in numchar:\n",
    "        if (character==i):\n",
    "            flag=1\n",
    "            break\n",
    "        else:\n",
    "            flag=0\n",
    "    if(flag==1):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "    \n",
    "    \n",
    "# double quotes\n",
    "\n",
    "def isDoubleQuotes(character):\n",
    "    if(character=='\"'):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "        \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#Abbreviation\n",
    "\n",
    "# Type 1 abbreviation. For eg. Dr., Mr. Mrs. etc.\n",
    "def is_abbr1(word):\n",
    "    flag=0\n",
    "    abbr1 = [\"Dr.\",\"Mr.\",\"Mrs.\",\"Phd.\",\"Prof.\",\"Drs.\",\"etc.\",\"i.e.\",\"eg.\",\"Rs.\"]\n",
    "    for i in range(len(abbr1)):\n",
    "#         print(abbr1[i]) \n",
    "        if(abbr1[i]==word):\n",
    "#             print(word)\n",
    "            flag=1\n",
    "    if(flag==1):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "        \n",
    "# Type 2 abbreviation. For eg. S.A.R. , R.A.D. etc.        \n",
    "def is_abbr2(word):\n",
    "    for i in range(len(word)):\n",
    "        if(word[i].isupper()) and (word[i+1]==\".\"):\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "\n",
    "        \n",
    "# is word capital\n",
    "\n",
    "def isWordCap(word):\n",
    "    if(word[0]).isupper():\n",
    "        if(len(word)==1):\n",
    "            return True\n",
    "        for i in range(1,len(word)):\n",
    "            if(word[i].islower()):\n",
    "                return True\n",
    "            else:\n",
    "                return False\n",
    "\n",
    "            \n",
    "# Words ending with double quotes\n",
    "def endQuotes(word):\n",
    "    n = len(word)\n",
    "    if(isDoubleQuotes(word[n-1]) and isTermPunct(word[n-2])):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "# Words starting with double quotes\n",
    "def startQuotes(word):\n",
    "    n = len(word)\n",
    "    if(isDoubleQuotes(word[0])):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "    \n",
    "    \n",
    "# to check urls\n",
    "def checkUrl(str):\n",
    " \n",
    "    # Regex to check valid URL\n",
    "    regex = (\"((http|https):https://)(www.)?\" +\n",
    "             \"[a-zA-Z0-9@:%._\\\\+~#?&//=]\" +\n",
    "             \"{2,256}\\\\.[a-z]\" +\n",
    "             \"{2,6}\\\\b([-a-zA-Z0-9@:%\" +\n",
    "             \"._\\\\+~#?&//=]*)\")\n",
    "     \n",
    "    # Compile the ReGex\n",
    "    p = re.compile(regex)\n",
    "\n",
    "    if (str == None):\n",
    "        return False\n",
    "\n",
    "    if(re.search(p, str)):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "# to check Emails\n",
    "def checkEmail(email):\n",
    "    regex = r'\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b'\n",
    " \n",
    "    if(re.fullmatch(regex, email)):\n",
    "        return True\n",
    " \n",
    "    else:\n",
    "        return False\n",
    "\n",
    "# check elipsis ...\n",
    "def checkElipsis(word):\n",
    "    \n",
    "    n = len(word)\n",
    "    if(word[n-1]==\".\" and word[n-2]==\".\" and word[n-3]==\".\"):\n",
    "        return True\n",
    "    if(isDoubleQuotes(word[n-1]) and word[n-2]==\".\" and word[n-3]==\".\" and word[n-4]==\".\"):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "    \n",
    "    \n",
    "# check decimal numbers    \n",
    "def checkDecimal(word):\n",
    "    r = re.compile(r\"^\\d*[.,]?\\d*$\")\n",
    "    if(r.match(word)):\n",
    "        return True\n",
    "    else:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9be925c",
   "metadata": {},
   "source": [
    "## Rules function\n",
    "\n",
    "making a rules function to check all the rules applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dee79459",
   "metadata": {},
   "outputs": [],
   "source": [
    "def checkRules(strlist):\n",
    "    flag=0\n",
    "#     print(strlist[2])\n",
    "#     print(isWordCap(strlist[2]))\n",
    "    if (isTermPunct(strlist[1]) and (is_abbr1(strlist[0]))==False): # left not abbr + terminating punct + Word cap\n",
    "        if(isWordCap(strlist[2]) or strlist[2]=='$$'): #if right word is cap or $$(end word)\n",
    "            flag=1\n",
    "    if(endQuotes(strlist[0])) and (isWordCap(strlist[2])):   #leftword + term punct + end quotes + capital word \n",
    "        flag=1\n",
    "    if(isTermPunct(strlist[1])) and (startQuotes(strlist[2])):   #leftword + termpunct + startquotes\n",
    "        flag=1\n",
    "    if(checkElipsis(strlist[0]) and isWordCap(strlist[2])):     #leftword with elipsis + punct + rightword cap/small -> no\n",
    "        flag=1\n",
    "    if(checkUrl(strlist[0]) or checkEmail(strlist[0])):     #checking url and email in string\n",
    "        flag=0\n",
    "    if(checkDecimal(strlist[0])):    #checking decimal\n",
    "        flag=0    \n",
    "#     print(flag)\n",
    "    if(flag):\n",
    "        return True\n",
    "    else:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623ee8ab",
   "metadata": {},
   "source": [
    "## Sentance helper functions\n",
    "\n",
    "making sentance helper functions like:\n",
    "* splitting the paragraph to list for making initial list based on **Terminating punctuations** \n",
    "* converting list to strings\n",
    "* initial splitter of string which combines boith the above functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2379d1ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "## split paragraph to list\n",
    "\n",
    "def splitParaList(string):\n",
    "    final=[]\n",
    "    lis = []\n",
    "    s = string.split(\" \")\n",
    "    n=len(s)\n",
    "    for i in range(len(s)):\n",
    "#         print(s[i])\n",
    "        for j in s[i]:\n",
    "            if(isTermPunct(j)):\n",
    "                lis.append(s[i])\n",
    "                lis.append(j)\n",
    "                s[i] = s[i]+\"$$\"\n",
    "                if(i==(len(s)-1)):\n",
    "                    lis.append(\"$$\")\n",
    "                else:\n",
    "                    lis.append(s[i+1])\n",
    "\n",
    "    return lis, s\n",
    "\n",
    "\n",
    "## convert list to string\n",
    "def listToString(s): \n",
    "    \n",
    "    # initialize an empty string\n",
    "    str1 = \"\" \n",
    "    \n",
    "    # traverse in the string  \n",
    "    for i in range(len(s)): \n",
    "        if(i!=len(s)-1):\n",
    "            str1 += s[i]+\" \" \n",
    "        else:\n",
    "            str1+=s[i]\n",
    "    \n",
    "    # return string  \n",
    "    return str1 \n",
    "\n",
    "## Splitting the initial list based on terminating punctuations\n",
    "\n",
    "def initialSplitter(string):\n",
    "    finalis, finalstr = splitParaList(string)    \n",
    "    n=3\n",
    "    final = [finalis[i * n:(i + 1) * n] for i in range((len(finalis) + n - 1) // n )] \n",
    "    return final\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "671be628",
   "metadata": {},
   "source": [
    "## Checking sentance boundary\n",
    "\n",
    "* first we find the boundary by giving the initial strings list as input\n",
    "* we get output of the boundary words \n",
    "* then we split the final sentance based on the boundary and add sentance markers '$$'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "747c38f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "## finding sentance boundary\n",
    "\n",
    "def findBoundary(strlist):\n",
    "    list2 = []\n",
    "    for i in final:\n",
    "        if(checkRules(i)):\n",
    "            list2.append(i)\n",
    "    boundary = []\n",
    "    for i in list2:\n",
    "        boundary.append(i[0])\n",
    "        \n",
    "    return boundary\n",
    "\n",
    "## converting to sentance\n",
    "def finalSentance(string, boundary):\n",
    "    s = string.split(\" \")\n",
    "    n=len(s)\n",
    "    m=len(boundary)\n",
    "    for i in range(len(s)):\n",
    "        k=0\n",
    "        while(k<m):\n",
    "            if(s[i]==boundary[k]):\n",
    "                s[i]+=\"$$\"\n",
    "                break\n",
    "            k=k+1\n",
    "    return listToString(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f83ff50f",
   "metadata": {},
   "source": [
    "## Checking special cases\n",
    "\n",
    "checking boundary and special cases like:\n",
    "* email addresses\n",
    "* decimals\n",
    "* elipsis\n",
    "* urls\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1e4ef727",
   "metadata": {},
   "outputs": [],
   "source": [
    "newstr = 'My email address is joyeetadey@gmail.com. This S.A.R. is a very 56.8% good place. \"I found it... on website www.travel.org\", she said. \"Isn\\'t it nice?\"'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "eaaeace1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['joyeetadey@gmail.com.', '.', 'This'], ['joyeetadey@gmail.com.$$', '.', 'This'], ['S.A.R.', '.', 'is'], ['S.A.R.$$', '.', 'is'], ['S.A.R.$$$$', '.', 'is'], ['56.8%', '.', 'good'], ['place.', '.', '\"I'], ['it...', '.', 'on'], ['it...$$', '.', 'on'], ['it...$$$$', '.', 'on'], ['www.travel.org\",', '.', 'she'], ['www.travel.org\",$$', '.', 'she'], ['said.', '.', '\"Isn\\'t'], ['nice?\"', '?', '$$']] \n",
      "\n",
      "['joyeetadey@gmail.com.', 'joyeetadey@gmail.com.$$', 'place.', 'said.', 'nice?\"'] \n",
      "\n",
      "My email address is joyeetadey@gmail.com.$$ This S.A.R. is a very 56.8% good place.$$ \"I found it... on website www.travel.org\", she said.$$ \"Isn't it nice?\"$$ \n",
      "\n",
      "My email address is joyeetadey@gmail.com.\n",
      " This S.A.R. is a very 56.8% good place.\n",
      " \"I found it... on website www.travel.org\", she said.\n",
      " \"Isn't it nice?\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "final = initialSplitter(newstr)\n",
    "print(final,\"\\n\")\n",
    "bd = findBoundary(final)\n",
    "print(bd,\"\\n\")\n",
    "sent = finalSentance(newstr,bd)\n",
    "print(sent,\"\\n\")\n",
    "slist = sent.split(\"$$\")\n",
    "for j in slist:\n",
    "    print(j)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e74d3fc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = \"Mrs. Ferrars has never yet mentioned Edward's name, which does not surprise us; but, to our great astonishment, not a line has been received from him on the occasion. Perhaps, however, he is kept silent by his fear of offending, and I shall, therefore, give him a hint, by a line to Oxford, that his sister and I both think a letter of proper submission from him, addressed perhaps to Fanny, and by her shewn to her mother, might not be taken amiss; for we all know the tenderness of Mrs. Ferrars's heart, and that she wishes for nothing so much as to be on good terms with her children.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "29ff19cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['Mrs.', '.', 'Ferrars'], ['occasion.', '.', 'Perhaps,'], ['Mrs.', '.', \"Ferrars's\"], ['children.', '.', '$$']] \n",
      "\n",
      "['occasion.', 'children.'] \n",
      "\n",
      "Mrs. Ferrars has never yet mentioned Edward's name, which does not surprise us; but, to our great astonishment, not a line has been received from him on the occasion.$$ Perhaps, however, he is kept silent by his fear of offending, and I shall, therefore, give him a hint, by a line to Oxford, that his sister and I both think a letter of proper submission from him, addressed perhaps to Fanny, and by her shewn to her mother, might not be taken amiss; for we all know the tenderness of Mrs. Ferrars's heart, and that she wishes for nothing so much as to be on good terms with her children.$$ \n",
      "\n",
      "Mrs. Ferrars has never yet mentioned Edward's name, which does not surprise us; but, to our great astonishment, not a line has been received from him on the occasion.\n",
      " Perhaps, however, he is kept silent by his fear of offending, and I shall, therefore, give him a hint, by a line to Oxford, that his sister and I both think a letter of proper submission from him, addressed perhaps to Fanny, and by her shewn to her mother, might not be taken amiss; for we all know the tenderness of Mrs. Ferrars's heart, and that she wishes for nothing so much as to be on good terms with her children.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "final = initialSplitter(sample)\n",
    "print(final,\"\\n\")\n",
    "bd = findBoundary(final)\n",
    "print(bd,\"\\n\")\n",
    "sent = finalSentance(sample,bd)\n",
    "print(sent,\"\\n\")\n",
    "slist = sent.split(\"$$\")\n",
    "for j in slist:\n",
    "    print(j)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "908f3062",
   "metadata": {},
   "source": [
    "## Testing with Dataset\n",
    "\n",
    "we checked 50 paragraphs and splitting into sentances by importing the austen.txt dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "70e05db6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Paragraphs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>THIS little work was finished in the year 1803...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No one who had ever seen Catherine Morland in ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Such was Catherine Morland at ten. At fifteen,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mrs. Morland was a very good woman, and wished...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From Thompson, that he is very good. \"I wish I...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          Paragraphs\n",
       "0  THIS little work was finished in the year 1803...\n",
       "1  No one who had ever seen Catherine Morland in ...\n",
       "2  Such was Catherine Morland at ten. At fifteen,...\n",
       "3  Mrs. Morland was a very good woman, and wished...\n",
       "4  From Thompson, that he is very good. \"I wish I..."
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('austen_dataset.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f80c6b00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "THIS little work was finished in the year 1803, and intended for immediate publication.$$ It was disposed of to a bookseller, it was even advertised, and why the business proceeded no farther, the author has never been able to learn.$$ That any bookseller should think it worth-while to purchase what he did not think it worth-while to publish seems extraordinary.$$ But with this, neither the author nor the public have any other concern than as some observation is necessary upon those parts of the work which thirteen years have made comparatively obsolete.$$ The public are entreated to bear in mind that thirteen years have passed since it was finished, many more since it was begun, and that during that period, places, manners, books, and opinions have undergone considerable changes.$$\n",
      "\n",
      "\n",
      "1\n",
      "No one who had ever seen Catherine Morland in her infancy would have supposed her born to be an heroine.$$ Her situation in life, the character of her father and mother, her own person and disposition, were all equally against her.$$ Her father was a clergyman, without being neglected, or poor, and a very respectable man, though his name was Richard--and he had never been handsome.$$ He had a considerable independence besides two good livings--and he was not in the least addicted to locking up his daughters.$$ Her mother was a woman of useful plain sense, with a good temper, and, what is more remarkable, with a good constitution.$$ She had three sons before Catherine was born; and instead of dying in bringing the latter into the world, as anybody might expect, she still lived on--lived to have six children more--to see them growing up around her, and to enjoy excellent health herself.$$ A family of ten children will be always called a fine family, where there are heads and arms and legs enough for the number; but the Morlands had little other right to the word, for they were in general very plain, and Catherine, for many years of her life, as plain as any.$$ She had a thin awkward figure, a sallow skin without colour, dark lank hair, and strong features--so much for her person; and not less unpropitious for heroism seemed her mind.$$ She was fond of all boy's plays, and greatly preferred cricket not merely to dolls, but to the more heroic enjoyments of infancy, nursing a dormouse, feeding a canary-bird, or watering a rose-bush.$$ Indeed she had no taste for a garden; and if she gathered flowers at all, it was chiefly for the pleasure of mischief--at least so it was conjectured from her always preferring those which she was forbidden to take.$$ Such were her propensities--her abilities were quite as extraordinary.$$ She never could learn or understand anything before she was taught; and sometimes not even then, for she was often inattentive, and occasionally stupid.$$ Her mother was three months in teaching her only to repeat the “Beggar's Petition”; and after all, her next sister, Sally, could say it better than she did.$$ Not that Catherine was always stupid--by no means; she learnt the fable of “The Hare and Many Friends” as quickly as any girl in England.$$ Her mother wished her to learn\n",
      "\n",
      "\n",
      "2\n",
      "Such was Catherine Morland at ten.$$ At fifteen, appearances were mending; she began to curl her hair and long for balls; her complexion improved, her features were softened by plumpness and colour, her eyes gained more animation, and her figure more consequence.$$ Her love of dirt gave way to an inclination for finery, and she grew clean as she grew smart; she had now the pleasure of sometimes hearing her father and mother remark on her personal improvement. “Catherine grows quite a good-looking girl--she is almost pretty today,” were words which caught her ears now and then; and how welcome were the sounds!$$ To look almost pretty is an acquisition of higher delight to a girl who has been looking plain the first fifteen years of her life than a beauty from her cradle can ever receive.$$\n",
      "\n",
      "\n",
      "3\n",
      "Mrs. Morland was a very good woman, and wished to see her children everything they ought to be; but her time was so much occupied in lying-in and teaching the little ones, that her elder daughters were inevitably left to shift for themselves; and it was not very wonderful that Catherine, who had by nature nothing heroic about her, should prefer cricket, baseball, riding on horseback, and running about the country at the age of fourteen, to books--or at least books of information--for, provided that nothing like useful knowledge could be gained from them, provided they were all story and no reflection, she had never any objection to books at all.$$ But from fifteen to seventeen she was in training for a heroine; she read all such works as heroines must read to supply their memories with those quotations which are so serviceable and so soothing in the vicissitudes of their eventful lives.$$\n",
      "\n",
      "\n",
      "4\n",
      "From Thompson, that he is very good.$$ \"I wish I could be like that!\"$$\n",
      "\n",
      "\n",
      "5\n",
      "And from Shakespeare she gained a great store of information--amongst the rest.$$ And that a young woman in love always looks.$$\n",
      "\n",
      "\n",
      "6\n",
      "So far her improvement was sufficient--and in many other points she came on exceedingly well; for though she could not write sonnets, she brought herself to read them; and though there seemed no chance of her throwing a whole party into raptures by a prelude on the pianoforte, of her own composition, she could listen to other people's performance with very little fatigue.$$ Her greatest deficiency was in the pencil--she had no notion of drawing--not enough even to attempt a sketch of her lover's profile, that she might be detected in the design.$$ There she fell miserably short of the true heroic height.$$ At present she did not know her own poverty, for she had no lover to portray.$$ She had reached the age of seventeen, without having seen one amiable youth who could call forth her sensibility, without having inspired one real passion, and without having excited even any admiration but what was very moderate and very transient.$$ This was strange indeed!$$ But strange things may be generally accounted for if their cause be fairly searched out.$$ There was not one lord in the neighbourhood; no--not even a baronet.$$ There was not one family among their acquaintance who had reared and supported a boy accidentally found at their door--not one young man whose origin was unknown.$$ Her father had no ward, and the squire of the parish no children.$$\n",
      "\n",
      "\n",
      "7\n",
      "But when a young lady is to be a heroine, the perverseness of forty surrounding families cannot prevent her.$$ Something must and will happen to throw a hero in her way.$$\n",
      "\n",
      "\n",
      "8\n",
      "Mr. Allen, who owned the chief of the property about Fullerton, the village in Wiltshire where the Morlands lived, was ordered to Bath for the benefit of a gouty constitution--and his lady, a good-humoured woman, fond of Miss Morland, and probably aware that if adventures will not befall a young lady in her own village, she must seek them abroad, invited her to go with them.$$ Mr. and Mrs. Morland were all compliance, and Catherine all happiness.$$\n",
      "\n",
      "\n",
      "9\n",
      "In addition to what has been already said of Catherine Morland's personal and mental endowments, when about to be launched into all the difficulties and dangers of a six weeks' residence in Bath, it may be stated, for the reader's more certain information, lest the following pages should otherwise fail of giving any idea of what her character is meant to be, that her heart was affectionate; her disposition cheerful and open, without conceit or affectation of any kind--her manners just removed from the awkwardness and shyness of a girl; her person pleasing, and, when in good looks, pretty--and her mind about as ignorant and uninformed as the female mind at seventeen usually is.$$\n",
      "\n",
      "\n",
      "10\n",
      "When the hour of departure drew near, the maternal anxiety of Mrs. Morland will be naturally supposed to be most severe.$$ A thousand alarming presentiments of evil to her beloved Catherine from this terrific separation must oppress her heart with sadness, and drown her in tears for the last day or two of their being together; and advice of the most important and applicable nature must of course flow from her wise lips in their parting conference in her closet.$$ Cautions against the violence of such noblemen and baronets as delight in forcing young ladies away to some remote farm-house, must, at such a moment, relieve the fulness of her heart.$$ Who would not think so?$$ But Mrs. Morland knew so little of lords and baronets, that she entertained no notion of their general mischievousness, and was wholly unsuspicious of danger to her daughter from their machinations.$$ Her cautions were confined to the following points. “I beg, Catherine, you will always wrap yourself up very warm about the throat, when you come from the rooms at night; and I wish you would try to keep some account of the money you spend; I will give you this little book on purpose.”$$\n",
      "\n",
      "\n",
      "11\n",
      "Sally, or rather Sarah (for what young lady of common gentility will reach the age of sixteen without altering her name as far as she can?), must from situation be at this time the intimate friend and confidante of her sister.$$ It is remarkable, however, that she neither insisted on Catherine's writing by every post, nor exacted her promise of transmitting the character of every new acquaintance, nor a detail of every interesting conversation that Bath might produce.$$ Everything indeed relative to this important journey was done, on the part of the Morlands, with a degree of moderation and composure, which seemed rather consistent with the common feelings of common life, than with the refined susceptibilities, the tender emotions which the first separation of a heroine from her family ought always to excite.$$ Her father, instead of giving her an unlimited order on his banker, or even putting an hundred pounds bank-bill into her hands, gave her only ten guineas, and promised her more when she wanted it.$$\n",
      "\n",
      "\n",
      "12\n",
      "Under these unpromising auspices, the parting took place, and the journey began.$$ It was performed with suitable quietness and uneventful safety.$$ Neither robbers nor tempests befriended them, nor one lucky overturn to introduce them to the hero.$$ Nothing more alarming occurred than a fear, on Mrs. Allen's side, of having once left her clogs behind her at an inn, and that fortunately proved to be groundless.$$\n",
      "\n",
      "\n",
      "13\n",
      "They arrived at Bath.$$ Catherine was all eager delight--her eyes were here, there, everywhere, as they approached its fine and striking environs, and afterwards drove through those streets which conducted them to the hotel.$$ She was come to be happy, and she felt happy already.$$\n",
      "\n",
      "\n",
      "14\n",
      "They were soon settled in comfortable lodgings in Pulteney Street.$$\n",
      "\n",
      "\n",
      "15\n",
      "It is now expedient to give some description of Mrs. Allen, that the reader may be able to judge in what manner her actions will hereafter tend to promote the general distress of the work, and how she will, probably, contribute to reduce poor Catherine to all the desperate wretchedness of which a last volume is capable--whether by her imprudence, vulgarity, or jealousy--whether by intercepting her letters, ruining her character, or turning her out of doors.$$\n",
      "\n",
      "\n",
      "16\n",
      "Mrs. Allen was one of that numerous class of females, whose society can raise no other emotion than surprise at there being any men in the world who could like them well enough to marry them.$$ She had neither beauty, genius, accomplishment, nor manner.$$ The air of a gentlewoman, a great deal of quiet, inactive good temper, and a trifling turn of mind were all that could account for her being the choice of a sensible, intelligent man like Mr. Allen.$$ In one respect she was admirably fitted to introduce a young lady into public, being as fond of going everywhere and seeing everything herself as any young lady could be.$$ Dress was her passion.$$ She had a most harmless delight in being fine; and our heroine's entree into life could not take place till after three or four days had been spent in learning what was mostly worn, and her chaperone was provided with a dress of the newest fashion.$$ Catherine too made some purchases herself, and when all these matters were arranged, the important evening came which was to usher her into the Upper Rooms.$$ Her hair was cut and dressed by the best hand, her clothes put on with care, and both Mrs. Allen and her maid declared she looked quite as she should do.$$ With such encouragement, Catherine hoped at least to pass uncensured through the crowd.$$ As for admiration, it was always very welcome when it came, but she did not depend on it.$$\n",
      "\n",
      "\n",
      "17\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mrs. Allen was so long in dressing that they did not enter the ballroom till late.$$ The season was full, the room crowded, and the two ladies squeezed in as well as they could.$$ As for Mr. Allen, he repaired directly to the card-room, and left them to enjoy a mob by themselves.$$ With more care for the safety of her new gown than for the comfort of her protegee, Mrs. Allen made her way through the throng of men by the door, as swiftly as the necessary caution would allow; Catherine, however, kept close at her side, and linked her arm too firmly within her friend's to be torn asunder by any common effort of a struggling assembly.$$ But to her utter amazement she found that to proceed along the room was by no means the way to disengage themselves from the crowd; it seemed rather to increase as they went on, whereas she had imagined that when once fairly within the door, they should easily find seats and be able to watch the dances with perfect convenience.$$ But this was far from being the case, and though by unwearied diligence they gained even the top of the room, their situation was just the same; they saw nothing of the dancers but the high feathers of some of the ladies.$$ Still they moved on--something better was yet in view; and by a continued exertion of strength and ingenuity they found themselves at last in the passage behind the highest bench.$$ Here there was something less of crowd than below; and hence Miss Morland had a comprehensive view of all the company beneath her, and of all the dangers of her late passage through them.$$ It was a splendid sight, and she began, for the first time that evening, to feel herself at a ball: she longed to dance, but she had not an acquaintance in the room.$$ Mrs. Allen did all that she could do in such a case by saying very placidly, every now and then, “I wish you could dance, my dear--I wish you could get a partner.”$$ For some time her young friend felt obliged to her for these wishes; but they were repeated so often, and proved so totally ineffectual, that Catherine grew tired at last, and would thank her no more.$$\n",
      "\n",
      "\n",
      "18\n",
      "They were not long able, however, to enjoy the repose of the eminence they had so laboriously gained.$$ Everybody was shortly in motion for tea, and they must squeeze out like the rest.$$ Catherine began to feel something of disappointment--she was tired of being continually pressed against by people, the generality of whose faces possessed nothing to interest, and with all of whom she was so wholly unacquainted that she could not relieve the irksomeness of imprisonment by the exchange of a syllable with any of her fellow captives; and when at last arrived in the tea-room, she felt yet more the awkwardness of having no party to join, no acquaintance to claim, no gentleman to assist them.$$ They saw nothing of Mr. Allen; and after looking about them in vain for a more eligible situation, were obliged to sit down at the end of a table, at which a large party were already placed, without having anything to do there, or anybody to speak to, except each other.$$\n",
      "\n",
      "\n",
      "19\n",
      "Mrs. Allen congratulated herself, as soon as they were seated, on having preserved her gown from injury. “It would have been very shocking to have it torn,” said she, “would not it?$$ It is such a delicate muslin.$$ For my part I have not seen anything I like so well in the whole room, I assure you.”$$\n",
      "\n",
      "\n",
      "20\n",
      "How uncomfortable it is, whispered Catherine, not to have a single acquaintance here!$$ Yes, my dear,” replied Mrs. Allen, with perfect serenity, it is very uncomfortable indeed.$$\n",
      "\n",
      "\n",
      "21\n",
      "What shall we do?$$ The gentlemen and ladies at this table look as if they wondered why we came here--we seem forcing ourselves into their party.$$ \"Aye, so we do.$$ That is very disagreeable.$$ I wish we had a large acquaintance here.\"$$\n",
      "\n",
      "\n",
      "22\n",
      "\"Very true, my dear; and if we knew anybody we would join them directly.$$ The Skinners were here last year--I wish they were here now.\"�$$ Had not we better go away as it is?$$ Here are no tea-things for us, you see.$$\n",
      "\n",
      "\n",
      "23\n",
      "No more there are, indeed.$$ How very provoking!$$ But I think we had better sit still, for one gets so tumbled in such a crowd!$$ How is my head, my dear?$$ Somebody gave me a push that has hurt it, I am afraid.$$\n",
      "\n",
      "\n",
      "24\n",
      "No, indeed, it looks very nice.$$ But, dear Mrs. Allen, are you sure there is nobody you know in all this multitude of people?$$ I think you must know somebody.$$\n",
      "\n",
      "\n",
      "25\n",
      "I don't, upon my word--I wish I did.$$ I wish I had a large acquaintance here with all my heart, and then I should get you a partner.$$ I should be so glad to have you dance.$$ There goes a strange-looking woman!$$ What an odd gown she has got on!$$ How old-fashioned it is!$$ Look at the back.$$\n",
      "\n",
      "\n",
      "26\n",
      "After some time they received an offer of tea from one of their neighbours; it was thankfully accepted, and this introduced a light conversation with the gentleman who offered it, which was the only time that anybody spoke to them during the evening, till they were discovered and joined by Mr. Allen when the dance was over.$$\n",
      "\n",
      "\n",
      "27\n",
      "\"Well, Miss Morland\", said he, directly, \"I hope you have had an agreeable ball.\"$$ Very agreeable indeed,she replied, vainly endeavouring to hide a great yawn.$$\n",
      "\n",
      "\n",
      "28\n",
      "\"I wish she had been able to dance\", said his wife; I wish we could have got a partner for her.$$ I have been saying how glad I should be if the Skinners were here this winter instead of last; or if the Parrys had come, as they talked of once, she might have danced with George Parry.$$ \"I am so sorry she has not had a partner!\"$$\n",
      "\n",
      "\n",
      "29\n",
      "“We shall do better another evening I hope,” was Mr. Allen's consolation.$$\n",
      "\n",
      "\n",
      "30\n",
      "The company began to disperse when the dancing was over--enough to leave space for the remainder to walk about in some comfort; and now was the time for a heroine, who had not yet played a very distinguished part in the events of the evening, to be noticed and admired.$$ Every five minutes, by removing some of the crowd, gave greater openings for her charms.$$ She was now seen by many young men who had not been near her before.$$ Not one, however, started with rapturous wonder on beholding her, no whisper of eager inquiry ran round the room, nor was she once called a divinity by anybody.$$ Yet Catherine was in very good looks, and had the company only seen her three years before, they would now have thought her exceedingly handsome.$$\n",
      "\n",
      "\n",
      "31\n",
      "She was looked at, however, and with some admiration; for, in her own hearing, two gentlemen pronounced her to be a pretty girl.$$ Such words had their due effect; she immediately thought the evening pleasanter than she had found it before--her humble vanity was contented--she felt more obliged to the two young men for this simple praise than a true-quality heroine would have been for fifteen sonnets in celebration of her charms, and went to her chair in good humour with everybody, and perfectly satisfied with her share of public attention.$$\n",
      "\n",
      "\n",
      "32\n",
      "Every morning now brought its regular duties--shops were to be visited; some new part of the town to be looked at; and the pump-room to be attended, where they paraded up and down for an hour, looking at everybody and speaking to no one.$$ The wish of a numerous acquaintance in Bath was still uppermost with Mrs. Allen, and she repeated it after every fresh proof, which every morning brought, of her knowing nobody at all.$$\n",
      "\n",
      "\n",
      "33\n",
      "They made their appearance in the Lower Rooms; and here fortune was more favourable to our heroine.$$ The master of the ceremonies introduced to her a very gentlemanlike young man as a partner; his name was Tilney.$$ He seemed to be about four or five and twenty, was rather tall, had a pleasing countenance, a very intelligent and lively eye, and, if not quite handsome, was very near it.$$ His address was good, and Catherine felt herself in high luck.$$ There was little leisure for speaking while they danced; but when they were seated at tea, she found him as agreeable as she had already given him credit for being.$$ He talked with fluency and spirit--and there was an archness and pleasantry in his manner which interested, though it was hardly understood by her.$$ After chatting some time on such matters as naturally arose from the objects around them, he suddenly addressed her with--“I have hitherto been very remiss, madam, in the proper attentions of a partner here; I have not yet asked you how long you have been in Bath; whether you were ever here before; whether you have been at the Upper Rooms, the theatre, and the concert; and how you like the place altogether.$$ I have been very negligent--but are you now at leisure to satisfy me in these particulars?$$ If you are I will begin directly.”$$\n",
      "\n",
      "\n",
      "34\n",
      "“You need not give yourself that trouble, sir.”$$\n",
      "\n",
      "\n",
      "35\n",
      "“No trouble, I assure you, madam.”$$ Then forming his features into a set smile, and affectedly softening his voice, he added, with a simpering air, “Have you been long in Bath, madam?”$$\n",
      "\n",
      "\n",
      "36\n",
      "“About a week, sir,” replied Catherine, trying not to laugh.$$\n",
      "\n",
      "\n",
      "37\n",
      "“Really!” with affected astonishment.$$\n",
      "\n",
      "\n",
      "38\n",
      "“Why should you be surprised, sir?”$$\n",
      "\n",
      "\n",
      "39\n",
      "“Why, indeed!” said he, in his natural tone. “But some emotion must appear to be raised by your reply, and surprise is more easily assumed, and not less reasonable than any other.$$ Now let us go on.$$ Were you never here before, madam?”$$\n",
      "\n",
      "\n",
      "40\n",
      "“Indeed!$$ Have you yet honoured the Upper Rooms?”$$\n",
      "\n",
      "\n",
      "41\n",
      "“Yes, sir, I was there last Monday.”$$\n",
      "\n",
      "\n",
      "42\n",
      "“Have you been to the theatre?”$$\n",
      "\n",
      "\n",
      "43\n",
      "“Yes, sir, I was at the play on Tuesday.”$$\n",
      "\n",
      "\n",
      "44\n",
      "“Yes, sir, on Wednesday.”$$\n",
      "\n",
      "\n",
      "45\n",
      "“And are you altogether pleased with Bath?”$$\n",
      "\n",
      "\n",
      "46\n",
      "“Yes--I like it very well.”$$\n",
      "\n",
      "\n",
      "47\n",
      "“Now I must give one smirk, and then we may be rational again.” laugh. “I see what you think of me,” said he gravely--“I shall make but a poor figure in your journal tomorrow.”$$\n",
      "\n",
      "\n",
      "48\n",
      "“Yes, I know exactly what you will say: Friday, went to the Lower Rooms; wore my sprigged muslin robe with blue trimmings--plain black shoes--appeared to much advantage; but was strangely harassed by a queer, half-witted man, who would make me dance with him, and distressed me by his nonsense.”$$\n",
      "\n",
      "\n",
      "49\n",
      "“Indeed I shall say no such thing.”$$\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# str = 'Such was Catherine Morland at ten. At fifteen, appearances were mending; she began to curl her hair and long for balls; her complexion improved, her features were softened by plumpness and colour, her eyes gained more animation, and her figure more consequence. Her love of dirt gave way to an inclination for finery, and she grew clean as she grew smart; she had now the pleasure of sometimes hearing her father and mother remark on her personal improvement. \"Catherine grows quite a good-looking girl--she is almost pretty today,\" were words which caught her ears now and then; and how welcome were the sounds! To look almost pretty is an acquisition of higher delight to a girl who has been looking plain the first fifteen years of her life than a beauty from her cradle can ever receive.' \n",
    "lisstr=[]\n",
    "# df_res = pd.DataFrame(columns = ['givenPara', 'boundary_list', 'result'])\n",
    "# df_res\n",
    "# Creating an empty dictionary\n",
    "myDict = {}\n",
    "output = pd.DataFrame(columns = ['givenPara', 'boundary_list'])\n",
    "for k in range(50):\n",
    "    print(k)\n",
    "    str= df['Paragraphs'].iloc[k]\n",
    "    lisstr.append(str)\n",
    "    for i in lisstr:\n",
    "        final = initialSplitter(str)\n",
    "#         print(final)\n",
    "\n",
    "        bd = findBoundary(final)\n",
    "#         print(bd)\n",
    "\n",
    "        sent = finalSentance(str,bd)\n",
    "#         df_res.(['boundary_list'],axis=1) = sent\n",
    "        slist = sent.split(\"$$\")\n",
    "#         for j in slist:\n",
    "#             print(j)\n",
    "    print(sent)\n",
    "    myDict['givenPara'] = str\n",
    "    myDict['boundary_list']= sent\n",
    "    output = output.append(myDict, ignore_index=True)\n",
    "\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9d5a058",
   "metadata": {},
   "source": [
    "## Calculating score manually\n",
    "\n",
    "we manually checked for the 50 paragraphs for accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3d6dd826",
   "metadata": {},
   "outputs": [],
   "source": [
    "testresult = [0,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,0,1,1,1,1,1,0,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1]\n",
    "actual = [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "63d7449c",
   "metadata": {},
   "outputs": [],
   "source": [
    "output['result']= testresult\n",
    "output['actual']= actual"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e702c4c",
   "metadata": {},
   "source": [
    "## Ouput dataframe and accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a60329c9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>givenPara</th>\n",
       "      <th>boundary_list</th>\n",
       "      <th>result</th>\n",
       "      <th>actual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>THIS little work was finished in the year 1803...</td>\n",
       "      <td>THIS little work was finished in the year 1803...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No one who had ever seen Catherine Morland in ...</td>\n",
       "      <td>No one who had ever seen Catherine Morland in ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Such was Catherine Morland at ten. At fifteen,...</td>\n",
       "      <td>Such was Catherine Morland at ten.$$ At fiftee...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mrs. Morland was a very good woman, and wished...</td>\n",
       "      <td>Mrs. Morland was a very good woman, and wished...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From Thompson, that he is very good. \"I wish I...</td>\n",
       "      <td>From Thompson, that he is very good.$$ \"I wish...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>And from Shakespeare she gained a great store ...</td>\n",
       "      <td>And from Shakespeare she gained a great store ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>So far her improvement was sufficient--and in ...</td>\n",
       "      <td>So far her improvement was sufficient--and in ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>But when a young lady is to be a heroine, the ...</td>\n",
       "      <td>But when a young lady is to be a heroine, the ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mr. Allen, who owned the chief of the property...</td>\n",
       "      <td>Mr. Allen, who owned the chief of the property...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>In addition to what has been already said of C...</td>\n",
       "      <td>In addition to what has been already said of C...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>When the hour of departure drew near, the mate...</td>\n",
       "      <td>When the hour of departure drew near, the mate...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Sally, or rather Sarah (for what young lady of...</td>\n",
       "      <td>Sally, or rather Sarah (for what young lady of...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Under these unpromising auspices, the parting ...</td>\n",
       "      <td>Under these unpromising auspices, the parting ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>They arrived at Bath. Catherine was all eager ...</td>\n",
       "      <td>They arrived at Bath.$$ Catherine was all eage...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>They were soon settled in comfortable lodgings...</td>\n",
       "      <td>They were soon settled in comfortable lodgings...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>It is now expedient to give some description o...</td>\n",
       "      <td>It is now expedient to give some description o...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Mrs. Allen was one of that numerous class of f...</td>\n",
       "      <td>Mrs. Allen was one of that numerous class of f...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Mrs. Allen was so long in dressing that they d...</td>\n",
       "      <td>Mrs. Allen was so long in dressing that they d...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>They were not long able, however, to enjoy the...</td>\n",
       "      <td>They were not long able, however, to enjoy the...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Mrs. Allen congratulated herself, as soon as t...</td>\n",
       "      <td>Mrs. Allen congratulated herself, as soon as t...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>How uncomfortable it is, whispered Catherine, ...</td>\n",
       "      <td>How uncomfortable it is, whispered Catherine, ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>What shall we do? The gentlemen and ladies at ...</td>\n",
       "      <td>What shall we do?$$ The gentlemen and ladies a...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>\"Very true, my dear; and if we knew anybody we...</td>\n",
       "      <td>\"Very true, my dear; and if we knew anybody we...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>No more there are, indeed. How very provoking!...</td>\n",
       "      <td>No more there are, indeed.$$ How very provokin...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>No, indeed, it looks very nice. But, dear Mrs....</td>\n",
       "      <td>No, indeed, it looks very nice.$$ But, dear Mr...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>I don't, upon my word--I wish I did. I wish I ...</td>\n",
       "      <td>I don't, upon my word--I wish I did.$$ I wish ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>After some time they received an offer of tea ...</td>\n",
       "      <td>After some time they received an offer of tea ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>\"Well, Miss Morland\", said he, directly, \"I ho...</td>\n",
       "      <td>\"Well, Miss Morland\", said he, directly, \"I ho...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>\"I wish she had been able to dance\", said his ...</td>\n",
       "      <td>\"I wish she had been able to dance\", said his ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>“We shall do better another evening I hope,” w...</td>\n",
       "      <td>“We shall do better another evening I hope,” w...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>The company began to disperse when the dancing...</td>\n",
       "      <td>The company began to disperse when the dancing...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>She was looked at, however, and with some admi...</td>\n",
       "      <td>She was looked at, however, and with some admi...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Every morning now brought its regular duties--...</td>\n",
       "      <td>Every morning now brought its regular duties--...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>They made their appearance in the Lower Rooms;...</td>\n",
       "      <td>They made their appearance in the Lower Rooms;...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>“You need not give yourself that trouble, sir.”</td>\n",
       "      <td>“You need not give yourself that trouble, sir.”$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>“No trouble, I assure you, madam.” Then formin...</td>\n",
       "      <td>“No trouble, I assure you, madam.”$$ Then form...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>“About a week, sir,” replied Catherine, trying...</td>\n",
       "      <td>“About a week, sir,” replied Catherine, trying...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>“Really!” with affected astonishment.</td>\n",
       "      <td>“Really!” with affected astonishment.$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>“Why should you be surprised, sir?”</td>\n",
       "      <td>“Why should you be surprised, sir?”$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>“Why, indeed!” said he, in his natural tone. “...</td>\n",
       "      <td>“Why, indeed!” said he, in his natural tone. “...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>“Indeed! Have you yet honoured the Upper Rooms?”</td>\n",
       "      <td>“Indeed!$$ Have you yet honoured the Upper Roo...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>“Yes, sir, I was there last Monday.”</td>\n",
       "      <td>“Yes, sir, I was there last Monday.”$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>“Have you been to the theatre?”</td>\n",
       "      <td>“Have you been to the theatre?”$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>“Yes, sir, I was at the play on Tuesday.”</td>\n",
       "      <td>“Yes, sir, I was at the play on Tuesday.”$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>“Yes, sir, on Wednesday.”</td>\n",
       "      <td>“Yes, sir, on Wednesday.”$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>“And are you altogether pleased with Bath?”</td>\n",
       "      <td>“And are you altogether pleased with Bath?”$$</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>“Yes--I like it very well.”</td>\n",
       "      <td>“Yes--I like it very well.”$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>“Now I must give one smirk, and then we may be...</td>\n",
       "      <td>“Now I must give one smirk, and then we may be...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>“Yes, I know exactly what you will say: Friday...</td>\n",
       "      <td>“Yes, I know exactly what you will say: Friday...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>“Indeed I shall say no such thing.”</td>\n",
       "      <td>“Indeed I shall say no such thing.”$$</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            givenPara  \\\n",
       "0   THIS little work was finished in the year 1803...   \n",
       "1   No one who had ever seen Catherine Morland in ...   \n",
       "2   Such was Catherine Morland at ten. At fifteen,...   \n",
       "3   Mrs. Morland was a very good woman, and wished...   \n",
       "4   From Thompson, that he is very good. \"I wish I...   \n",
       "5   And from Shakespeare she gained a great store ...   \n",
       "6   So far her improvement was sufficient--and in ...   \n",
       "7   But when a young lady is to be a heroine, the ...   \n",
       "8   Mr. Allen, who owned the chief of the property...   \n",
       "9   In addition to what has been already said of C...   \n",
       "10  When the hour of departure drew near, the mate...   \n",
       "11  Sally, or rather Sarah (for what young lady of...   \n",
       "12  Under these unpromising auspices, the parting ...   \n",
       "13  They arrived at Bath. Catherine was all eager ...   \n",
       "14  They were soon settled in comfortable lodgings...   \n",
       "15  It is now expedient to give some description o...   \n",
       "16  Mrs. Allen was one of that numerous class of f...   \n",
       "17  Mrs. Allen was so long in dressing that they d...   \n",
       "18  They were not long able, however, to enjoy the...   \n",
       "19  Mrs. Allen congratulated herself, as soon as t...   \n",
       "20  How uncomfortable it is, whispered Catherine, ...   \n",
       "21  What shall we do? The gentlemen and ladies at ...   \n",
       "22  \"Very true, my dear; and if we knew anybody we...   \n",
       "23  No more there are, indeed. How very provoking!...   \n",
       "24  No, indeed, it looks very nice. But, dear Mrs....   \n",
       "25  I don't, upon my word--I wish I did. I wish I ...   \n",
       "26  After some time they received an offer of tea ...   \n",
       "27  \"Well, Miss Morland\", said he, directly, \"I ho...   \n",
       "28  \"I wish she had been able to dance\", said his ...   \n",
       "29  “We shall do better another evening I hope,” w...   \n",
       "30  The company began to disperse when the dancing...   \n",
       "31  She was looked at, however, and with some admi...   \n",
       "32  Every morning now brought its regular duties--...   \n",
       "33  They made their appearance in the Lower Rooms;...   \n",
       "34    “You need not give yourself that trouble, sir.”   \n",
       "35  “No trouble, I assure you, madam.” Then formin...   \n",
       "36  “About a week, sir,” replied Catherine, trying...   \n",
       "37              “Really!” with affected astonishment.   \n",
       "38                “Why should you be surprised, sir?”   \n",
       "39  “Why, indeed!” said he, in his natural tone. “...   \n",
       "40   “Indeed! Have you yet honoured the Upper Rooms?”   \n",
       "41               “Yes, sir, I was there last Monday.”   \n",
       "42                    “Have you been to the theatre?”   \n",
       "43          “Yes, sir, I was at the play on Tuesday.”   \n",
       "44                          “Yes, sir, on Wednesday.”   \n",
       "45        “And are you altogether pleased with Bath?”   \n",
       "46                        “Yes--I like it very well.”   \n",
       "47  “Now I must give one smirk, and then we may be...   \n",
       "48  “Yes, I know exactly what you will say: Friday...   \n",
       "49                “Indeed I shall say no such thing.”   \n",
       "\n",
       "                                        boundary_list  result  actual  \n",
       "0   THIS little work was finished in the year 1803...       0       1  \n",
       "1   No one who had ever seen Catherine Morland in ...       1       1  \n",
       "2   Such was Catherine Morland at ten.$$ At fiftee...       1       1  \n",
       "3   Mrs. Morland was a very good woman, and wished...       1       1  \n",
       "4   From Thompson, that he is very good.$$ \"I wish...       1       1  \n",
       "5   And from Shakespeare she gained a great store ...       1       1  \n",
       "6   So far her improvement was sufficient--and in ...       1       1  \n",
       "7   But when a young lady is to be a heroine, the ...       1       1  \n",
       "8   Mr. Allen, who owned the chief of the property...       1       1  \n",
       "9   In addition to what has been already said of C...       0       1  \n",
       "10  When the hour of departure drew near, the mate...       1       1  \n",
       "11  Sally, or rather Sarah (for what young lady of...       1       1  \n",
       "12  Under these unpromising auspices, the parting ...       1       1  \n",
       "13  They arrived at Bath.$$ Catherine was all eage...       1       1  \n",
       "14  They were soon settled in comfortable lodgings...       1       1  \n",
       "15  It is now expedient to give some description o...       1       1  \n",
       "16  Mrs. Allen was one of that numerous class of f...       1       1  \n",
       "17  Mrs. Allen was so long in dressing that they d...       1       1  \n",
       "18  They were not long able, however, to enjoy the...       0       1  \n",
       "19  Mrs. Allen congratulated herself, as soon as t...       1       1  \n",
       "20  How uncomfortable it is, whispered Catherine, ...       1       1  \n",
       "21  What shall we do?$$ The gentlemen and ladies a...       1       1  \n",
       "22  \"Very true, my dear; and if we knew anybody we...       1       1  \n",
       "23  No more there are, indeed.$$ How very provokin...       1       1  \n",
       "24  No, indeed, it looks very nice.$$ But, dear Mr...       0       1  \n",
       "25  I don't, upon my word--I wish I did.$$ I wish ...       1       1  \n",
       "26  After some time they received an offer of tea ...       1       1  \n",
       "27  \"Well, Miss Morland\", said he, directly, \"I ho...       1       1  \n",
       "28  \"I wish she had been able to dance\", said his ...       1       1  \n",
       "29  “We shall do better another evening I hope,” w...       1       1  \n",
       "30  The company began to disperse when the dancing...       1       1  \n",
       "31  She was looked at, however, and with some admi...       1       1  \n",
       "32  Every morning now brought its regular duties--...       1       1  \n",
       "33  They made their appearance in the Lower Rooms;...       0       1  \n",
       "34  “You need not give yourself that trouble, sir.”$$       1       1  \n",
       "35  “No trouble, I assure you, madam.”$$ Then form...       1       1  \n",
       "36  “About a week, sir,” replied Catherine, trying...       1       1  \n",
       "37            “Really!” with affected astonishment.$$       1       1  \n",
       "38              “Why should you be surprised, sir?”$$       1       1  \n",
       "39  “Why, indeed!” said he, in his natural tone. “...       1       1  \n",
       "40  “Indeed!$$ Have you yet honoured the Upper Roo...       1       1  \n",
       "41             “Yes, sir, I was there last Monday.”$$       1       1  \n",
       "42                  “Have you been to the theatre?”$$       1       1  \n",
       "43        “Yes, sir, I was at the play on Tuesday.”$$       1       1  \n",
       "44                        “Yes, sir, on Wednesday.”$$       1       1  \n",
       "45      “And are you altogether pleased with Bath?”$$       0       1  \n",
       "46                      “Yes--I like it very well.”$$       1       1  \n",
       "47  “Now I must give one smirk, and then we may be...       1       1  \n",
       "48  “Yes, I know exactly what you will say: Friday...       1       1  \n",
       "49              “Indeed I shall say no such thing.”$$       1       1  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "88e3ce6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9\n"
     ]
    }
   ],
   "source": [
    "## accuracy of the model \n",
    "## checking manually 45/50\n",
    "accuracy  = 0.90\n",
    "print(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a18ed481",
   "metadata": {},
   "source": [
    "## Observations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68dd7122",
   "metadata": {},
   "source": [
    "* In most of the cases, we get a wrong result if the font style of the input sentance has been changed.\n",
    "* Mostly, the errors concern with double quotaion marks.\n",
    "* Single quotation marks has not been considered\n",
    "* The sentance given in input should be grammatically correct to get proper answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02f4cfeb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}