Covid19-QnA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objective:\n",
    "\n",
    "We develop a simple Question and Answer System using bert model for covid-19 queries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing the required modules and frameworks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from transformers import BertTokenizer, TFBertForQuestionAnswering \n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "gpus = tf.config.experimental.list_physical_devices('GPU')\n",
    "tf.config.experimental.set_memory_growth(gpus[0], True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Taking a pretrained model\n",
    "\n",
    "We will be taking the pretrained model of bert from the hugging face website. We take the large bert model because we need good accuracy for answering the question that ha been provided since it is related to medical fied.<br> \n",
    "\n",
    "Tokenizer is used to encode the query and sentences for the model to understand.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All model checkpoint weights were used when initializing TFBertForQuestionAnswering.\n",
      "\n",
      "All the weights of TFBertForQuestionAnswering were initialized from the model checkpoint at bert-large-uncased-whole-word-masking-finetuned-squad.\n",
      "If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFBertForQuestionAnswering for predictions without further training.\n"
     ]
    }
   ],
   "source": [
    "modelName = 'bert-large-uncased-whole-word-masking-finetuned-squad' # https://huggingface.co/transformers/pretrained_models.html\n",
    "#modelName = 'distilbert-base-cased-distilled-squad'\n",
    "tokenizer = BertTokenizer.from_pretrained(modelName)\n",
    "model = TFBertForQuestionAnswering.from_pretrained(modelName)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get source and query\n",
    "\n",
    "We will be taking Covid-19 information from the wikipedia text as the truth source for our model.\n",
    "Link: https://en.wikipedia.org/wiki/Coronavirus_disease_2019\n",
    "\n",
    "For a example query we will be taking a question such as **\"What are the symptoms of COVID-19?\"**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "source=r\"\"\"The COVID‑19 pandemic, also known as the coronavirus pandemic, is an ongoing global pandemic of coronavirus disease 2019 (COVID‑19). The outbreak was first identified in December 2019 in Wuhan, China. The World Health Organization declared the outbreak a Public Health Emergency of International Concern on 30 January 2020 and a pandemic on 11 March. As of 6 August 2020, more than 18.7 million cases of COVID‑19 have been reported in more than 188 countries and territories, resulting in more than 706,000 deaths; more than 11.3 million people have recovered.The virus is primarily spread between people during close contact,most often via small droplets produced by coughing,sneezing, and talking.The droplets usually fall to the ground or onto surfaces rather than travelling through air over long distances.However, the transmission may also occur through smaller droplets that are able to stay suspended in the air for longer periods of time in enclosed spaces, as typical for airborne diseases. Less commonly, people may become infected by touching a contaminated surface and then touching their face.It is most contagious during the first three days after the onset of symptoms, although spread is possible before symptoms appear, and from people who do not show symptoms.Common symptoms include fever, cough, fatigue, shortness of breath, and loss of sense of smell. Complications may include pneumonia and acute respiratory distress syndrome.The time from exposure to onset of symptoms is typically around five days but may range from two to fourteen days.There is no known vaccine or specific antiviral treatment.Primary treatment is symptomatic and supportive therapy.Recommended preventive measures include hand washing, covering one's mouth when coughing, maintaining distance from other people, wearing a face mask in public settings, disinfecting surfaces, increasing ventilation and air filtration indoors, and monitoring and self-isolation for people who suspect they are infected.Authorities worldwide have responded by implementing travel restrictions, lockdowns, workplace hazard controls, and facility closures in order to slow the spread of the disease. Many places have also worked to increase testing capacity and trace contacts of infected persons.The pandemic has caused global social and economic disruption, global famines affecting 265 million people.\"\"\"\n",
    "question =r\"\"\"What are the symptoms of COVID-19?\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocess them to pass into a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add a seperator such as **\" [SEP] \"** between the source and the question for the model to know. Then we encode the input text with the tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_text=question+\" [SEP] \"+source\n",
    "input_ids=tokenizer.encode(input_text)\n",
    "input_1 = tf.constant(input_ids)[None, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[101, 2054, 2024, 1996, 8030, 1997, 2522, 17258, 1011, 2539, 1029, 102, 1996, 2522, 17258, 1514, 2539, 6090, 3207, 7712, 1010, 2036, 2124, 2004, 1996, 21887, 23350, 6090, 3207, 7712, 1010, 2003, 2019, 7552, 3795, 6090, 3207, 7712, 1997, 21887, 23350, 4295, 10476, 1006, 2522, 17258, 1514, 2539, 1007, 1012, 1996, 8293, 2001, 2034, 4453, 1999, 2285, 10476, 1999, 8814, 4819, 1010, 2859, 1012, 1996, 2088, 2740, 3029, 4161, 1996, 8293, 1037, 2270, 2740, 5057, 1997, 2248, 5142, 2006, 2382, 2254, 12609, 1998, 1037, 6090, 3207, 7712, 2006, 2340, 2233, 1012, 2004, 1997, 1020, 2257, 12609, 1010, 2062, 2084, 2324, 1012, 1021, 2454, 3572, 1997, 2522, 17258, 1514, 2539, 2031, 2042, 2988, 1999, 2062, 2084, 19121, 3032, 1998, 6500, 1010, 4525, 1999, 2062, 2084, 3963, 2575, 1010, 2199, 6677, 1025, 2062, 2084, 2340, 1012, 1017, 2454, 2111, 2031, 6757, 1012, 1996, 7865, 2003, 3952, 3659, 2090, 2111, 2076, 2485, 3967, 1010, 2087, 2411, 3081, 2235, 27126, 2550, 2011, 21454, 1010, 1055, 24045, 6774, 1010, 1998, 3331, 1012, 1996, 27126, 2788, 2991, 2000, 1996, 2598, 2030, 3031, 9972, 2738, 2084, 8932, 2083, 2250, 2058, 2146, 12103, 1012, 2174, 1010, 1996, 6726, 2089, 2036, 5258, 2083, 3760, 27126, 2008, 2024, 2583, 2000, 2994, 6731, 1999, 1996, 2250, 2005, 2936, 6993, 1997, 2051, 1999, 10837, 7258, 1010, 2004, 5171, 2005, 10519, 7870, 1012, 2625, 4141, 1010, 2111, 2089, 2468, 10372, 2011, 7244, 1037, 19450, 3302, 1998, 2059, 7244, 2037, 2227, 1012, 2009, 2003, 2087, 9530, 15900, 6313, 2076, 1996, 2034, 2093, 2420, 2044, 1996, 14447, 1997, 8030, 1010, 2348, 3659, 2003, 2825, 2077, 8030, 3711, 1010, 1998, 2013, 2111, 2040, 2079, 2025, 2265, 8030, 1012, 2691, 8030, 2421, 9016, 1010, 19340, 1010, 16342, 1010, 2460, 2791, 1997, 3052, 1010, 1998, 3279, 1997, 3168, 1997, 5437, 1012, 12763, 2089, 2421, 18583, 1998, 11325, 16464, 12893, 8715, 1012, 1996, 2051, 2013, 7524, 2000, 14447, 1997, 8030, 2003, 4050, 2105, 2274, 2420, 2021, 2089, 2846, 2013, 2048, 2000, 7426, 2420, 1012, 2045, 2003, 2053, 2124, 17404, 2030, 3563, 3424, 24093, 2140, 3949, 1012, 3078, 3949, 2003, 25353, 27718, 9626, 4588, 1998, 16408, 7242, 1012, 6749, 4652, 3512, 5761, 2421, 2192, 12699, 1010, 5266, 2028, 1005, 1055, 2677, 2043, 21454, 1010, 8498, 3292, 2013, 2060, 2111, 1010, 4147, 1037, 2227, 7308, 1999, 2270, 10906, 1010, 4487, 11493, 25969, 2075, 9972, 1010, 4852, 19536, 1998, 2250, 10882, 7096, 8156, 24274, 1010, 1998, 8822, 1998, 2969, 1011, 12477, 2005, 2111, 2040, 8343, 2027, 2024, 10372, 1012, 4614, 4969, 2031, 5838, 2011, 14972, 3604, 9259, 1010, 5843, 7698, 2015, 1010, 16165, 15559, 7711, 1010, 1998, 4322, 8503, 2015, 1999, 2344, 2000, 4030, 1996, 3659, 1997, 1996, 4295, 1012, 2116, 3182, 2031, 2036, 2499, 2000, 3623, 5604, 3977, 1998, 7637, 10402, 1997, 10372, 5381, 1012, 1996, 6090, 3207, 7712, 2038, 3303, 3795, 2591, 1998, 3171, 20461, 1010, 3795, 15625, 2015, 12473, 20549, 2454, 2111, 1012, 102]\n",
      "[CLS] what are the symptoms of covid - 19? [SEP] the covid ‑ 19 pandemic, also known as the coronavirus pandemic, is an ongoing global pandemic of coronavirus disease 2019 ( covid ‑ 19 ). the outbreak was first identified in december 2019 in wuhan, china. the world health organization declared the outbreak a public health emergency of international concern on 30 january 2020 and a pandemic on 11 march. as of 6 august 2020, more than 18. 7 million cases of covid ‑ 19 have been reported in more than 188 countries and territories, resulting in more than 706, 000 deaths ; more than 11. 3 million people have recovered. the virus is primarily spread between people during close contact, most often via small droplets produced by coughing, sneezing, and talking. the droplets usually fall to the ground or onto surfaces rather than travelling through air over long distances. however, the transmission may also occur through smaller droplets that are able to stay suspended in the air for longer periods of time in enclosed spaces, as typical for airborne diseases. less commonly, people may become infected by touching a contaminated surface and then touching their face. it is most contagious during the first three days after the onset of symptoms, although spread is possible before symptoms appear, and from people who do not show symptoms. common symptoms include fever, cough, fatigue, shortness of breath, and loss of sense of smell. complications may include pneumonia and acute respiratory distress syndrome. the time from exposure to onset of symptoms is typically around five days but may range from two to fourteen days. there is no known vaccine or specific antiviral treatment. primary treatment is symptomatic and supportive therapy. recommended preventive measures include hand washing, covering one's mouth when coughing, maintaining distance from other people, wearing a face mask in public settings, disinfecting surfaces, increasing ventilation and air filtration indoors, and monitoring and self - isolation for people who suspect they are infected. authorities worldwide have responded by implementing travel restrictions, lockdowns, workplace hazard controls, and facility closures in order to slow the spread of the disease. many places have also worked to increase testing capacity and trace contacts of infected persons. the pandemic has caused global social and economic disruption, global famines affecting 265 million people. [SEP]\n"
     ]
    }
   ],
   "source": [
    "print(input_ids)\n",
    "print(tokenizer.decode(input_ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation:** From the above we can see how the input text is encoded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n"
     ]
    }
   ],
   "source": [
    "token_ids=[0 if i < input_ids.index(102) else 1 for i in range(len(input_ids))]\n",
    "print(token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation:** We will be using token_ids to give importance to the source by giving the source tokens a value of 1 and question token a value of 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prediction from our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer=model({'input_ids':input_1,'token_type_ids': tf.convert_to_tensor([token_ids])})\n",
    "startScores,endScores=answer\n",
    "input_tokens = tokenizer.convert_ids_to_tokens(input_ids)\n",
    "startIdx = tf.math.argmax(startScores[0],0).numpy()\n",
    "endIdx = tf.math.argmax(endScores[0],0).numpy()+1\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process(st):\n",
    "    new=st.split('#')\n",
    "    new_st=\"\"\n",
    "    #print(new)\n",
    "    #print(len(new))\n",
    "    for word in new:\n",
    "        if word==\"\":\n",
    "            new_st=new_st[:-1]\n",
    "        new_st+=word\n",
    "    return new_st\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fever , cough , fatigue , shortness of breath , and loss of sense of smell\n"
     ]
    }
   ],
   "source": [
    "ans=process(\" \".join(input_tokens[startIdx:endIdx]))\n",
    "print(ans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation:** we can see it is giving accurate results for the symptoms of covid-19."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Utility function:\n",
    "We will be developing a utility function so that this could be integrated anywhere for covid-19 QnA system:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def covid_19_qna(question):\n",
    "    # Load the pretrained model\n",
    "    modelName = 'bert-large-uncased-whole-word-masking-finetuned-squad' # https://huggingface.co/transformers/pretrained_models.html\n",
    "    tokenizer = BertTokenizer.from_pretrained(modelName)\n",
    "    model = TFBertForQuestionAnswering.from_pretrained(modelName)\n",
    "    # Covid-19 Information\n",
    "    source=r\"\"\"The COVID‑19 pandemic, also known as the coronavirus pandemic, is an ongoing global pandemic of coronavirus disease 2019 (COVID‑19). The outbreak was first identified in December 2019 in Wuhan, China. The World Health Organization declared the outbreak a Public Health Emergency of International Concern on 30 January 2020 and a pandemic on 11 March. As of 6 August 2020, more than 18.7 million cases of COVID‑19 have been reported in more than 188 countries and territories, resulting in more than 706,000 deaths; more than 11.3 million people have recovered.The virus is primarily spread between people during close contact,most often via small droplets produced by coughing,sneezing, and talking.The droplets usually fall to the ground or onto surfaces rather than travelling through air over long distances.However, the transmission may also occur through smaller droplets that are able to stay suspended in the air for longer periods of time in enclosed spaces, as typical for airborne diseases. Less commonly, people may become infected by touching a contaminated surface and then touching their face.It is most contagious during the first three days after the onset of symptoms, although spread is possible before symptoms appear, and from people who do not show symptoms.Common symptoms include fever, cough, fatigue, shortness of breath, and loss of sense of smell. Complications may include pneumonia and acute respiratory distress syndrome.The time from exposure to onset of symptoms is typically around five days but may range from two to fourteen days.There is no known vaccine or specific antiviral treatment.Primary treatment is symptomatic and supportive therapy.Recommended preventive measures include hand washing, covering one's mouth when coughing, maintaining distance from other people, wearing a face mask in public settings, disinfecting surfaces, increasing ventilation and air filtration indoors, and monitoring and self-isolation for people who suspect they are infected.Authorities worldwide have responded by implementing travel restrictions, lockdowns, workplace hazard controls, and facility closures in order to slow the spread of the disease. Many places have also worked to increase testing capacity and trace contacts of infected persons.The pandemic has caused global social and economic disruption, global famines affecting 265 million people.\"\"\"\n",
    "    # Concatiing and preprocessing for our bert model\n",
    "    input_text=question+\" [SEP] \"+source\n",
    "    input_ids=tokenizer.encode(input_text)\n",
    "    input_1 = tf.constant(input_ids)[None, :]\n",
    "    # Token Ids\n",
    "    token_ids=[0 if i < input_ids.index(102) else 1 for i in range(len(input_ids))]\n",
    "    # Model prediction\n",
    "    answer=model({'input_ids':input_1,'token_type_ids': tf.convert_to_tensor([token_ids])})\n",
    "    startScores,endScores=answer\n",
    "    input_tokens = tokenizer.convert_ids_to_tokens(input_ids)\n",
    "    startIdx = tf.math.argmax(startScores[0],0).numpy()\n",
    "    endIdx = tf.math.argmax(endScores[0],0).numpy()+1\n",
    "    # [!] Note: we need process function from the above\n",
    "    ans=process(\" \".join(input_tokens[startIdx:endIdx]))\n",
    "    return ans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your query for Covid-19: Where has covid-19 first detected ?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All model checkpoint weights were used when initializing TFBertForQuestionAnswering.\n",
      "\n",
      "All the weights of TFBertForQuestionAnswering were initialized from the model checkpoint at bert-large-uncased-whole-word-masking-finetuned-squad.\n",
      "If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFBertForQuestionAnswering for predictions without further training.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wuhan , china\n"
     ]
    }
   ],
   "source": [
    "query=input(\"Enter your query for Covid-19: \")\n",
    "print(covid_19_qna(query))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation:** We can see that the query gave us correct results and this function can be integrated anywhere for QnA of Covid-19 System."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}