notebooks/Information Retrieval - NLP Part III BERT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples and questions for NLP using BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML, IFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Start by watching the following two videos that introduce Neural Networks (optional) and  BERT (run the cell to embed the videos in this notebook) then answer the questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"560\"\n",
       "            height=\"315\"\n",
       "            src=\"https://www.youtube.com/embed/fkqZyYo_ebs?rel=0&amp;controls=0&amp;showinfo=0\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f75e8e54fd0>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## OPTIONAL Intro to NN\n",
    "IFrame(src=\"https://www.youtube.com/embed/fkqZyYo_ebs?rel=0&amp;controls=0&amp;showinfo=0\", width=\"560\", height=\"315\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"560\"\n",
       "            height=\"315\"\n",
       "            src=\"https://www.youtube.com/embed/xI0HHN5XKDo?rel=0&amp;controls=0&amp;showinfo=0\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f75e8e4ee90>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## REQUIRED Intro to BERT\n",
    "IFrame(\"https://www.youtube.com/embed/xI0HHN5XKDo?rel=0&amp;controls=0&amp;showinfo=0\", width=\"560\", height=\"315\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1: Name two issues with the LSTM recurrent neural networks that Transformer networks address"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2: How do the Transformer networks address the two issues?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 3: What are the two primary components of Transformer networks?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4: Which component of the Transfomer network is stacked to create a BERT network?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 5: What are 4 problems that the video mentions BERT networks can address?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 6: What are the 2 steps to solving problems with BERT? What is the primary goal of each of the steps?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 7: How does the Masked Language Model task help BERT to understand language?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 8: How does the predication task help BERT to understand language?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 9: What change to BERT network architecture is needed to fine tune?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 10: True/False - each input token to BERT is a human readable token representing a single word?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 11: In your own words, what is a WordPiece model (HINT: see [this blog post](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/#21-special-tokens)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 12: True/False - the output word vectors of a BERT model are generated sequentially?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 13 (BONUS): True/False - the loss function for BERT training is [cross-entropy](https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 14 (BONUS): True/False - the loss function of the model is calculated for masked words only in order to increase the network's attention to context?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 15 (BONUS):  True/False - the BERT base model has more than 300 million paramaters?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Exploring Transformer models using the Huggingface Transformer package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this vignette/assignment we are using Transformer models from the Transformer package (https://github.com/huggingface/transformers). I recommend that you visit the github page for the project and check out the various demos in the \"Online demos\" section. I have set up this Jupyter notebook so that you can import the transformers library and pytorch. This allows you to run several of the test cases from the libraries detailed documentation located at https://huggingface.co/transformers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2a. Masked language modeling example - this example is explained at https://huggingface.co/transformers/task_summary.html#masked-language-modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we import a helpful utility class called 'pipeline'. This class implements several common NLP workflows - see https://huggingface.co/transformers/main_classes/pipelines.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the fill-mask pipeline using a basic BERT model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = pipeline(\"fill-mask\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "pprint(nlp(f\"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Question 17: FOR YOU TO DO:  Replace the '...' in the following Python statement with a sentence of your own that uses the \n",
    "##### `{nlp.tokenizer.mask_token}` and then run the cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pprint(nlp(f\"... {nlp.tokenizer.mask_token} ...\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 18. True/False - Since all BERT models are trained using word masking, this model should perform as well on clinical statements as on statements about any other topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2b. An example that doesn't use the pipeline helper - this follows the example at https://huggingface.co/transformers/task_summary.html#named-entity-recognition "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next example uses a model and a tokenizer. Note that this example is using a specific BERT model called 'bert-base-cased'. You can learn more about this model here: https://huggingface.co/bert-base-cased\n",
    "\n",
    "After reading more about the bert-base-cased, take note that there are many other models available on the same site: https://huggingface.co/models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 19. Search the [models posted by the huggingface community](https://huggingface.co/models). Use tags dropdown and search form on the website. What are two models that you found that are possible interest to you? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Back to the example, as the [transformers website explains](https://huggingface.co/transformers/task_summary.html#named-entity-recognition ), the of process of using a transformer for NER is the following:\n",
    "\n",
    "1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it with the weights stored in the checkpoint.\n",
    "\n",
    "2. Define the label list with which the model was trained on.\n",
    "\n",
    "3. Define a sequence with known entities, such as “Hugging Face” as an organisation and “New York City” as a location.\n",
    "\n",
    "4. Split words into tokens so that they can be mapped to predictions. We use a small hack by, first, completely encoding and decoding the sequence, so that we’re left with a string that contains the special tokens.\n",
    "\n",
    "5. Encode that sequence into IDs (special tokens are added automatically).\n",
    "\n",
    "6. Retrieve the predictions by passing the input to the model and getting the first output. This results in a distribution over the 9 possible classes for each token. We take the argmax to retrieve the most likely class for each token.\n",
    "\n",
    "7. Zip together each token with its prediction and print it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForTokenClassification, AutoTokenizer\n",
    "import torch\n",
    "\n",
    "model = AutoModelForTokenClassification.from_pretrained(\"dbmdz/bert-large-cased-finetuned-conll03-english\", return_dict=True)\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n",
    "\n",
    "label_list = [\n",
    "    \"O\",       # Outside of a named entity\n",
    "    \"B-MISC\",  # Beginning of a miscellaneous entity right after another miscellaneous entity\n",
    "    \"I-MISC\",  # Miscellaneous entity\n",
    "    \"B-PER\",   # Beginning of a person's name right after another person's name\n",
    "    \"I-PER\",   # Person's name\n",
    "    \"B-ORG\",   # Beginning of an organisation right after another organisation\n",
    "    \"I-ORG\",   # Organisation\n",
    "    \"B-LOC\",   # Beginning of a location right after another location\n",
    "    \"I-LOC\"    # Location\n",
    "]\n",
    "\n",
    "sequence = \"Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very\" \\\n",
    "           \"close to the Manhattan Bridge.\"\n",
    "\n",
    "# Bit of a hack to get the tokens with the special tokens\n",
    "tokens = tokenizer.tokenize(tokenizer.decode(tokenizer.encode(sequence)))\n",
    "inputs = tokenizer.encode(sequence, return_tensors=\"pt\")\n",
    "\n",
    "outputs = model(inputs).logits\n",
    "predictions = torch.argmax(outputs, dim=2)\n",
    "print([(token, label_list[prediction]) for token, prediction in zip(tokens, predictions[0].numpy())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand how this example works, you need to understand how textual data is preprocessed the role of a `tokenizer`. Read these two pages and then answer the questions: \n",
    "\n",
    "a. [Preprocessing](https://huggingface.co/transformers/preprocessing.html)\n",
    "\n",
    "b. [Tokenizer](https://huggingface.co/transformers/tokenizer_summary.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 20: Why do you need to preprocess the sentences you want to pass to a BERT model using a tokenizer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 21: What problem to [sub-word tokenization algorithms](https://huggingface.co/transformers/tokenizer_summary.html) solve?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing fine tuned clinicalBERT and fine tuned generic BERT for Medical Natural Language Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you learned from the video, BERT models have a general understanding of language and should be fine-tuned to address specific NLP problems. In general, this process involves the following:\n",
    "\n",
    "1. Developing or acquiring a training/test set so that the model can be fine-tuned using supervised learning.\n",
    "\n",
    "2. Transforming the training/test set data to the format required by the BERT Transformer encoder.\n",
    "\n",
    "3. Configuring the hyperparameters for how the BERT model will learn during fine tuning.\n",
    "\n",
    "4. Training the model.\n",
    "\n",
    "5. Testing the model.\n",
    "\n",
    "I have gone through these steps for you for the Medical Natural Language Inference NLP problem. This problem involves inferring a clinical fact from the text of a clinical note. The training/test set I used was the [MedNLI](https://physionet.org/content/mednli/1.0.0/)[[1]] dataset created using MIMIC III notes [[2]]. That dataset has a number of sentences from clinical notes that have been labeled by clinicians for specific inferences about the patient's clinical status. Here are some examples:\n",
    "\n",
    "```\n",
    "[\n",
    " ('Labs were notable for Cr 1.7 (baseline 0.5 per old records) and lactate 2.4.', ' Patient has elevated Cr', 'entailment'), \n",
    " ('Labs were notable for Cr 1.7 (baseline 0.5 per old records) and lactate 2.4.', ' Patient has normal Cr', 'contradiction'), \n",
    " ('Labs were notable for Cr 1.7 (baseline 0.5 per old records) and lactate 2.4.', ' Patient has elevated BUN', 'neutral')\n",
    "]\n",
    "```\n",
    "I fine tuned two different BERT models:\n",
    "\n",
    "1. [clinicalBERT](https://github.com/EmilyAlsentzer/clinicalBERT) - read about it [here](https://www.aclweb.org/anthology/W19-1909/)[[3.]]\n",
    "\n",
    "2. [bert-base-cased](https://huggingface.co/bert-base-cased) - read about it [here](https://arxiv.org/abs/1810.04805)[[4.]]\n",
    "\n",
    "The code I used to fine tune the models was downloaded from here: https://github.com/EmilyAlsentzer/clinicalBERT/tree/master/downstream_tasks\n",
    "\n",
    "Fine tuning took less than 20 minutes for both BERT models using a server with a NVIDIA GLX 2080 Ti GPU. \n",
    "\n",
    "The cells below allow you to run both models on sample sentences. **Run the cells and then answer the questions that follow.**\n",
    "\n",
    "\n",
    "*References*\n",
    "\n",
    "[[1.]] Shivade, C. (2019). MedNLI - A Natural Language Inference Dataset For The Clinical Domain (version 1.0.0). PhysioNet. https://doi.org/10.13026/C2RS98.\n",
    "\n",
    "[[2.]] Goldberger, A., Amaral, L., Glass, L., Hausdorff, J., Ivanov, P. C., Mark, R., ... & Stanley, H. E. (2000). PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation [Online]. 101 (23), pp. e215–e220.\n",
    "\n",
    "[[3.]] Alsentzer, Emily, et al. \"Publicly available clinical BERT embeddings.\" arXiv preprint arXiv:1904.03323 (2019).\n",
    "\n",
    "[[4.]] Devlin, Jacob, et al. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\" arXiv preprint arXiv:1810.04805 (2018)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Sample sentences\n",
    "stplL = [\n",
    "    (\"In the ED, initial VS revealed T 98.9, HR 73, BP 121/90, RR 15, O2 sat 98% on RA.\",\" The patient is hemodynamically stable\"),\n",
    "    (\"Labs were notable for Cr 1.7 (baseline 0.5 per old records) and lactate 2.4.\",\" Patient has elevated Cr\"),\n",
    "    (\"Labs were notable for Cr 1.7 (baseline 0.5 per old records) and lactate 2.4.\",\" Patient has normal Cr\"),\n",
    "    (\"Labs were notable for Cr 1.7 (baseline 0.5 per old records) and lactate 2.4.\",\" Patient has elevated BUN\"),\n",
    "    ('No history of blood clots or DVTs, has never had chest pain prior to one week ago.', ' Patient has angina'), \n",
    "    ('No history of blood clots or DVTs, has never had chest pain prior to one week ago.', ' Patient has had multiple PEs'), \n",
    "    ('No history of blood clots or DVTs, has never had chest pain prior to one week ago.', ' Patient has CAD'),\n",
    "    ('In the ED, initial VS revealed T 98.9, HR 73, BP 121/90, RR 15, O2 sat 98% on RA.', ' The patient is hemodynamically stable '),\n",
    "    ('In the ED, initial VS revealed T 98.9, HR 73, BP 121/90, RR 15, O2 sat 98% on RA.', ' The patient is hemodynamically unstable'),\n",
    "    ('In the ED, initial VS revealed T 98.9, HR 73, BP 121/90, RR 15, O2 sat 98% on RA.', ' The patient is in pain.')      \n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## First test clinicalBERT\n",
    "from transformers import BertTokenizer, BertForSequenceClassification\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Imports the tokenizer that has the vocabulary (embeddings) and the fine tuned model\n",
    "tokenizer = BertTokenizer.from_pretrained(\"emilyalsentzer/Bio_ClinicalBERT\")\n",
    "model = BertForSequenceClassification.from_pretrained(\"../clinBertFineTunedMedNLI/\",num_labels=3,return_dict=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## The labels used during fine tuning\n",
    "label_list = [\"contradiction\",\"entailment\",\"neutral\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Iterates through the sentence pairs, tokenizes the sentences, passess the sentences \n",
    "## through the model, and obtains the highest scoring label (see label_list above) from the \n",
    "## logits output by the model \n",
    "for tpl in stplL:\n",
    "    (s1,s2) = tpl\n",
    "    inputs = tokenizer(s1, s2, add_special_tokens=True, return_tensors=\"pt\")\n",
    "    input_ids = inputs[\"input_ids\"].tolist()[0]\n",
    "    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)\n",
    "    labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1\n",
    "    outputs = model(**inputs, labels=labels)\n",
    "    loss = outputs.loss\n",
    "    logits = outputs.logits\n",
    "    predictions = torch.argmax(logits[0], dim=0)\n",
    "    \n",
    "    print(\"sentence 1: \" + s1)\n",
    "    print(\"sentence 2: \" + s2)\n",
    "    print(\"prediction: \" + label_list[predictions])\n",
    "    print(\"loss: \" + str(loss))\n",
    "    print()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## now, rerun all of the same steps above using the generic cased BERT that has not been fine tuned\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-cased')\n",
    "model = BertForSequenceClassification.from_pretrained('bert-base-cased', return_dict=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## The labels used during fine tuning\n",
    "label_list = [\"contradiction\",\"entailment\",\"neutral\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for tpl in stplL:\n",
    "    (s1,s2) = tpl\n",
    "    inputs = tokenizer(s1, s2, add_special_tokens=True, return_tensors=\"pt\")\n",
    "    input_ids = inputs[\"input_ids\"].tolist()[0]\n",
    "    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)\n",
    "    labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1\n",
    "    outputs = model(**inputs, labels=labels)\n",
    "    loss = outputs.loss\n",
    "    logits = outputs.logits\n",
    "    predictions = torch.argmax(logits[0], dim=0)\n",
    "    \n",
    "    print(\"sentence 1: \" + s1)\n",
    "    print(\"sentence 2: \" + s2)\n",
    "    print(\"prediction: \" + label_list[predictions])\n",
    "    print(\"loss: \" + str(loss))\n",
    "    print()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## Finally, rerun all of the same steps above using the generic cased BERT that HAS been fine tuned\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-cased')\n",
    "model = BertForSequenceClassification.from_pretrained(\"../BertCasedFineTunedMedNLI/\",num_labels=3,return_dict=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## The labels used during fine tuning\n",
    "label_list = [\"contradiction\",\"entailment\",\"neutral\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for tpl in stplL:\n",
    "    (s1,s2) = tpl\n",
    "    inputs = tokenizer(s1, s2, add_special_tokens=True, return_tensors=\"pt\")\n",
    "    input_ids = inputs[\"input_ids\"].tolist()[0]\n",
    "    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)\n",
    "    labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1\n",
    "    outputs = model(**inputs, labels=labels)\n",
    "    loss = outputs.loss\n",
    "    logits = outputs.logits\n",
    "    predictions = torch.argmax(logits[0], dim=0)\n",
    "    \n",
    "    print(\"sentence 1: \" + s1)\n",
    "    print(\"sentence 2: \" + s2)\n",
    "    print(\"prediction: \" + label_list[predictions])\n",
    "    print(\"loss: \" + str(loss))\n",
    "    print()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 22. What is your impression about which model(s) do better at clinical natural language inference? If so, can you offer a possible reason for why? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 22. Please describe how you would formally compare the performance of the three models using the MedNLI training/test set? Be specific about the metrics you would use and what criteria you would use to determine that one model is better than another?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 22. What role, if any, would inter-rater agreement statistics such as Kappa play in a formal evaluation?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 23:  In your opinion, what kinds of Biomedical NLP problems might BERT not be a good fit for, and why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook was created by Rich Boyce and William Reynolds with help from Sanya B. Taneja and NLP expert Denis R Newman-Griffis <dnewmangriffis@pitt.edu>. Contact Denis if you are interested in research in biomedical NLP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}