language/examples/prompt-design/question_answering.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ur8xi4C7S06n"
   },
   "outputs": [],
   "source": [
    "# Copyright 2023 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAPoU8Sm5E6e"
   },
   "source": [
    "# Question Answering with Generative Models on Vertex AI\n",
    "\n",
    "\n",
    "<table align=\"left\">\n",
    "\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/language/examples/prompt-design/question_answering.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/prompt-design/question_answering.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/prompt-design/question_answering.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      Open in Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "## Overview\n",
    "\n",
    "Large language models can be used for various natural language processing tasks, including question-answering (Q&A). These models are trained on a vast amount text data and can generate high-quality responses to a wide range of questions. One thing to note here is that most models have cutoff dates regarding their knowledge, and asking anything too recent might yield an incomplete, imaginative or incorrect answer (i.e. a hallucination).\n",
    "\n",
    "This notebook covers the essentials of prompts for answering questions using a generative model. In addition, it showcases the `open domain` (knowledge available on the public internet) and `closed domain` (knowledge that is more private - typically enterprise or personal knowledge).\n",
    "\n",
    "Learn more about prompt design in the [official documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/text/text-overview#prompt_structure)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d975e698c9a4"
   },
   "source": [
    "### Objective\n",
    "\n",
    "By the end of the notebook, you should be able to write prompts for the following:\n",
    "\n",
    "* **Open domain** questions:\n",
    "    * Zero-shot prompting\n",
    "    * Few-shot prompting\n",
    "\n",
    "\n",
    "* **Closed domain** questions:\n",
    "    * Providing custom knowledge as context\n",
    "    * Instruction-tune the outputs\n",
    "    * Few-shot prompting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QDU0XJ1xRDlL"
   },
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2a5AEr0lkLKD"
   },
   "source": [
    "### Install Vertex AI SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "82ad0c445061"
   },
   "outputs": [],
   "source": [
    "!pip install google-cloud-aiplatform --upgrade --user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab only:** Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_Hsqwn4hkLKE"
   },
   "outputs": [],
   "source": [
    "# # Automatically restart kernel after installs so that your environment can access the new packages\n",
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Xe7OuYuGkLKF"
   },
   "source": [
    "### Authenticating your notebook environment\n",
    "* If you are using **Colab** to run this notebook, uncomment the cell below and continue.\n",
    "* If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "U9Gx2SAZkLKF"
   },
   "outputs": [],
   "source": [
    "# from google.colab import auth\n",
    "# auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "960505627ddf"
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab only:** Uncomment the following cell to initialize the Vertex AI SDK. For Vertex AI Workbench, you don't need to run this.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import vertexai\n",
    "\n",
    "# PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
    "# vertexai.init(project=PROJECT_ID, location=\"us-central1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PyQmSRbKA8r-"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from vertexai.preview.language_models import TextGenerationModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UP76a2la7O-a"
   },
   "source": [
    "### Import models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7isig7e07O-a"
   },
   "outputs": [],
   "source": [
    "generation_model = TextGenerationModel.from_pretrained(\"text-bison@001\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fIPcn5dZ7O-b"
   },
   "source": [
    "## Question Answering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cNNEz7vGFYUP"
   },
   "source": [
    "Question-answering capabilities require providing a prompt or a question that the model can use to generate a response. The prompt can be a few words or a few complete sentences, depending on the complexity of the question.\n",
    "\n",
    "When creating a question-answering prompt, it is essential to be specific and provide as much context as possible. It helps the model understand the intent behind the question and generate a relevant response. For example, if you want to ask:\n",
    "\n",
    "```\n",
    "\"What is the capital of France?\",\n",
    "\n",
    "then a good prompt could be:\n",
    "\n",
    "\"Please tell me the name of the city that serves as the capital of France.\"\n",
    "\n",
    "```\n",
    "\n",
    "In addition to being specific, the prompt should also be grammatically correct and free of spelling errors. It helps the model generate a response that is easy to understand and contains fewer errors or inaccuracies.\n",
    "\n",
    "By providing specific, context-rich prompts, you can help the model understand the intent behind the question and generate accurate and relevant responses.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C5N9ZnlECm-z"
   },
   "source": [
    "Below are some differences between the **open domain** and **closed domain** categories for question-answering prompts.\n",
    "\n",
    "* **Open domain**: All questions whose answers are available online already. They can belong to any category, like history, geography, countries, politics, chemistry, etc. These include trivia or general knowledge questions, like:\n",
    "\n",
    "```\n",
    "Q: Who won the Olympic gold in swimming?\n",
    "Q: Who is the President of [given country]?\n",
    "Q: Who wrote [specific book]\"?\n",
    "```\n",
    "\n",
    "Keep in mind the training cutoff of generative models, as questions involving information more recent than what the model was trained on might give incorrect or imaginative answers.\n",
    "\n",
    "\n",
    "* **Closed domain**: If you have some internal knowledge base not available on the public internet, then those belong to the _closed domain_ category.\n",
    "You can pass that \"private\" knowledge as context to the model. If prompted correctly, the model is more likely to answer from within the context provided and less likely to give answers beyond that from the open internet.\n",
    "\n",
    "Consider the example of building a Q&A bot over your internal product documentation. In this case, you can pass the complete documentation to the model and prompt it only to answer based on that.\n",
    "\n",
    "Typical prompt for **closed domain**:\n",
    "\n",
    "```\n",
    "Prompt: f\"\"\" Answer from the below context: \\n\\n\n",
    "\t\t   context: {your knowledge base} \\n\n",
    "\t\t   question: {question specific to that knowledge base}  \\n\n",
    "\t\t   answer: {to be predicted by model} \\n\n",
    "\t\t\"\"\"\n",
    "```\n",
    "\n",
    "Below are some examples to understand these different types of prompts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WBoN6zixDSiX"
   },
   "source": [
    "### Open Domain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wJnv8XhnDXQm"
   },
   "source": [
    "#### Zero-shot prompting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PaYoQuRwCm-z"
   },
   "outputs": [],
   "source": [
    "prompt = \"\"\"Q: Who was President of the United States in 1955? Which party did he belong to?\\n\n",
    "            A:\n",
    "         \"\"\"\n",
    "print(\n",
    "    generation_model.predict(\n",
    "        prompt,\n",
    "        max_output_tokens=256,\n",
    "        temperature=0.1,\n",
    "    ).text\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4qcUdUgwCm-z"
   },
   "outputs": [],
   "source": [
    "prompt = \"\"\"Q: What is the tallest mountain in the world?\\n\n",
    "            A:\n",
    "         \"\"\"\n",
    "print(\n",
    "    generation_model.predict(\n",
    "        prompt,\n",
    "        max_output_tokens=20,\n",
    "        temperature=0.1,\n",
    "    ).text\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HShw52X2Dcmx"
   },
   "source": [
    "#### Few-shot prompting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tj_2hHAWE8vh"
   },
   "source": [
    "Let's say you want to a get a short answer from the model (like only a specific name). To do so, you can leverage a few-shot prompt and provide examples to the model to illustrate the expected behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RE5yCAaqDg7m"
   },
   "outputs": [],
   "source": [
    "prompt = \"\"\"Q: Who is the current President of France?\\n\n",
    "            A: Emmanuel Macron \\n\\n\n",
    "\n",
    "            Q: Who invented the telephone? \\n\n",
    "            A: Alexander Graham Bell \\n\\n\n",
    "\n",
    "            Q: Who wrote the novel \"1984\"?\n",
    "            A: George Orwell\n",
    "\n",
    "            Q: Who discovered penicillin?\n",
    "            A:\n",
    "         \"\"\"\n",
    "print(\n",
    "    generation_model.predict(\n",
    "        prompt,\n",
    "        max_output_tokens=20,\n",
    "        temperature=0.1,\n",
    "    ).text\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xGvs0jFsUlvM"
   },
   "source": [
    "#### Zero-shot prompting vs Few-shot prompting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7yjsAMuMUfZC"
   },
   "source": [
    "Zero-shot prompting can be useful for quickly generating text for new tasks, but the quality of the generated text may be lower than that of a few-shot prompt with well-chosen examples. Few-shot prompting is typically better suited for tasks that require a high degree of specificity or domain-specific knowledge, but requires some additional thought and potentially data to set up the prompt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u6UiJTxXEs4t"
   },
   "source": [
    "### Closed Domain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "03ZITm4AGBvP"
   },
   "source": [
    "#### Adding internal knowledge as context in prompts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EkhqjmB6VqPx"
   },
   "source": [
    "Imagine a scenario where you would like to build a question-answering bot that takes in internal documentation and lets users ask questions about it.\n",
    "\n",
    "In the example below, the Google Cloud Storage and content policy documentation is added to the prompt, so that the PaLM API can use that to answer subsequent questions with the provided context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "s1H2er_lExpW"
   },
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Storage and content policy \\n\n",
    "How durable is my data in Cloud Storage? \\n\n",
    "Cloud Storage is designed for 99.999999999% (11 9's) annual durability, which is appropriate for even primary storage and\n",
    "business-critical applications. This high durability level is achieved through erasure coding that stores data pieces redundantly\n",
    "across multiple devices located in multiple availability zones.\n",
    "Objects written to Cloud Storage must be redundantly stored in at least two different availability zones before the\n",
    "write is acknowledged as successful. Checksums are stored and regularly revalidated to proactively verify that the data\n",
    "integrity of all data at rest as well as to detect corruption of data in transit. If required, corrections are automatically\n",
    "made using redundant data. Customers can optionally enable object versioning to add protection against accidental deletion.\n",
    "\"\"\"\n",
    "\n",
    "question = \"How is high availability achieved?\"\n",
    "\n",
    "prompt = f\"\"\"Answer the question given in the contex below:\n",
    "Context: {context}?\\n\n",
    "Question: {question} \\n\n",
    "Answer:\n",
    "\"\"\"\n",
    "\n",
    "print(\"[Prompt]\")\n",
    "print(prompt)\n",
    "\n",
    "print(\"[Response]\")\n",
    "print(\n",
    "    generation_model.predict(\n",
    "        prompt,\n",
    "    ).text\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tagWC4VcQIw6"
   },
   "source": [
    "#### Instruction-tuning outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v9UkogWHXM6N"
   },
   "source": [
    "Another way to help out language models is to provide additional instructions to frame the output in the prompt. To ensure the model doesn't respond to anything outside the context, the prompt can specify that the response should be \"Information not available in provided context\" if that's the case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ouq8FfwSQIBT"
   },
   "outputs": [],
   "source": [
    "question = \"What machined are required for hosting Vertex AI models?\"\n",
    "prompt = f\"\"\"Answer the question given the context below as {{Context:}}. \\n\n",
    "If the answer is not available in the {{Context:}} and you are not confident about the output,\n",
    "please say \"Information not available in provided context\". \\n\\n\n",
    "Context: {context}?\\n\n",
    "Question: {question} \\n\n",
    "Answer:\n",
    "\"\"\"\n",
    "\n",
    "print(\"[Prompt]\")\n",
    "print(prompt)\n",
    "\n",
    "print(\"[Response]\")\n",
    "print(\n",
    "    generation_model.predict(\n",
    "        prompt,\n",
    "        max_output_tokens=256,\n",
    "        temperature=0.3,\n",
    "    ).text\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iZJfZShPRGqU"
   },
   "source": [
    "#### Few-shot prompting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qdSEQeQIS6pt"
   },
   "outputs": [],
   "source": [
    "prompt = \"\"\"\n",
    "Context:\n",
    "The term \"artificial intelligence\" was first coined by John McCarthy in 1956. Since then, AI has developed into a vast\n",
    "field with numerous applications, ranging from self-driving cars to virtual assistants like Siri and Alexa.\n",
    "\n",
    "Question:\n",
    "What is artificial intelligence?\n",
    "\n",
    "Answer:\n",
    "Artificial intelligence refers to the simulation of human intelligence in machines that are programmed to think and learn like humans.\n",
    "\n",
    "---\n",
    "\n",
    "Context:\n",
    "The Wright brothers, Orville and Wilbur, were two American aviation pioneers who are credited with inventing and\n",
    "building the world's first successful airplane and making the first controlled, powered and sustained heavier-than-air human flight,\n",
    " on December 17, 1903.\n",
    "\n",
    "Question:\n",
    "Who were the Wright brothers?\n",
    "\n",
    "Answer:\n",
    "The Wright brothers were American aviation pioneers who invented and built the world's first successful airplane\n",
    "and made the first controlled, powered and sustained heavier-than-air human flight, on December 17, 1903.\n",
    "\n",
    "---\n",
    "\n",
    "Context:\n",
    "The Mona Lisa is a 16th-century portrait painted by Leonardo da Vinci during the Italian Renaissance. It is one of\n",
    "the most famous paintings in the world, known for the enigmatic smile of the woman depicted in the painting.\n",
    "\n",
    "Question:\n",
    "Who painted the Mona Lisa?\n",
    "\n",
    "Answer:\n",
    "\n",
    "\"\"\"\n",
    "print(\n",
    "    generation_model.predict(\n",
    "        prompt,\n",
    "    ).text\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "leYIui80Q4tH"
   },
   "source": [
    "### Extractive Question-Answering\n",
    "\n",
    "In the next example, the generative model is guided to understand the meaning of the question and the passage, and to identify the relevant information in the passage that answers the question. The model is given a question and a passage of text, and is asked to find the answer to the question within the passage. The answer is typically a phrase or sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bPZqm0QJQ4tH"
   },
   "outputs": [],
   "source": [
    "prompt = \"\"\"\n",
    "Background: There is evidence that there have been significant changes in Amazon rainforest vegetation over the last 21,000 years through the Last Glacial Maximum (LGM) and subsequent deglaciation.\n",
    "Analyses of sediment deposits from Amazon basin paleo lakes and from the Amazon Fan indicate that rainfall in the basin during the LGM was lower than for the present, and this was almost certainly\n",
    "associated with reduced moist tropical vegetation cover in the basin. There is debate, however, over how extensive this reduction was. Some scientists argue that the rainforest was reduced to small,\n",
    "isolated refugia separated by open forest and grassland; other scientists argue that the rainforest remained largely intact but extended less far to the north, south, and east than is seen today.\n",
    "This debate has proved difficult to resolve because the practical limitations of working in the rainforest mean that data sampling is biased away from the center of the Amazon basin, and both\n",
    "explanations are reasonably well supported by the available data.\n",
    "\n",
    "Q: What does LGM stands for?\n",
    "A: Last Glacial Maximum.\n",
    "\n",
    "Q: What did the analysis from the sediment deposits indicate?\n",
    "A: Rainfall in the basin during the LGM was lower than for the present.\n",
    "\n",
    "Q: What are some of scientists arguments?\n",
    "A: The rainforest was reduced to small, isolated refugia separated by open forest and grassland.\n",
    "\n",
    "Q: There have been major changes in Amazon rainforest vegetation over the last how many years?\n",
    "A: 21,000.\n",
    "\n",
    "Q: What caused changes in the Amazon rainforest vegetation?\n",
    "A: The Last Glacial Maximum (LGM) and subsequent deglaciation\n",
    "\n",
    "Q: What has been analyzed to compare Amazon rainfall in the past and present?\n",
    "A: Sediment deposits.\n",
    "\n",
    "Q: What has the lower rainfall in the Amazon during the LGM been attributed to?\n",
    "A:\n",
    "\"\"\"\n",
    "\n",
    "print(\n",
    "    generation_model.predict(\n",
    "        prompt,\n",
    "    ).text\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "94d80fb55f48"
   },
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b620d23a7634"
   },
   "source": [
    "You can evaluate the outputs of the question and answering task if the ground truth answers of each question are available. In zero-shot prompting, you can only use `open domain` questions. However, with `closed domain` questions, you can add context and evaluate similarly.  To showcase how that will work, start by creating a simple dataframe with questions and ground truth answers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8e813a463531"
   },
   "outputs": [],
   "source": [
    "qa_data = {\n",
    "    \"question\": [\n",
    "        \"In a website browser address bar, what does “www” stand for?\",\n",
    "        \"Who was the first woman to win a Nobel Prize\",\n",
    "        \"What is the name of the Earth’s largest ocean?\",\n",
    "    ],\n",
    "    \"answer_groundtruth\": [\"World Wide Web\", \"Marie Curie\", \"The Pacific Ocean\"],\n",
    "}\n",
    "qa_data_df = pd.DataFrame(qa_data)\n",
    "qa_data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "951a147dc79d"
   },
   "source": [
    "Now that you have the data with questions and ground truth answers, you can call the PaLM 2 generation model to each review row using the `apply` function. Each row will use the dynamic prompt to predict the answer using the PaLM API. We will save the results in `answer_prediction` column.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ffc47e0cb5b9"
   },
   "outputs": [],
   "source": [
    "def get_answer(row):\n",
    "    prompt = f\"\"\"Answer the following question as precise as possible.\\n\\n\n",
    "            question: {row}\n",
    "            answer:\n",
    "              \"\"\"\n",
    "    return generation_model.predict(\n",
    "        prompt=prompt,\n",
    "    ).text\n",
    "\n",
    "\n",
    "qa_data_df[\"answer_prediction\"] = qa_data_df[\"question\"].apply(get_answer)\n",
    "qa_data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6fe997dbf788"
   },
   "source": [
    "You may want to evaluate the answers predicted by the PaLM API. However, it will be more complex than the text classification since the answers may differ from ground truth and may be presented in slightly more/fewer words. \n",
    "\n",
    "For example, you can observe the question \"What is the name of the Earth's largest ocean?\" and see that model predicted  \"Pacific Ocean\" when a ground truth label is \"The Pacific Ocean\" with the extra \"The.\" Now, if you use the simple classification metrics, then you will consider this as a wrong prediction since original and predicted strings have a difference. However, you can see that the answer is correct since an extra \"The\" is causing the issue. It's a simple string comparison problem.\n",
    "\n",
    "The solution to string comparison where both `ground_thruth` and `predicted` may have some extra or fewer letters, one approach is to use a fuzzy matching algorithm. \n",
    "Fuzzy string matching uses [Levenshtein Distance](https://en.wikipedia.org/wiki/Levenshtein_distance) to calculate the differences between two strings. \n",
    "\n",
    "For example, the Levenshtein distance between \"kitten\" and \"sitting\" is 3, since the following 3 edits change one into the other, and there is no way to do it with fewer than 3 edits:\n",
    "\n",
    "* kitten → sitten (substitution of \"s\" for \"k\"),\n",
    "* sitten → sittin (substitution of \"i\" for \"e\"),\n",
    "* sittin → sitting (insertion of \"g\" at the end).\n",
    "\n",
    "\n",
    "Here's another example, but this time using `fuzzywuzzy`  library, which gives us the same `Levenshtein distance` between two strings but in ratio. The ratio raw score measures the string's similarity as an int in the range [0, 100]. For two strings X and Y, the score is defined by int(round((2.0 * M / T) * 100)) where T is the total number of characters in both strings, and M is the number of matches in the two strings. \n",
    "\n",
    "Read more here about the [ratio formula](https://anhaidgroup.github.io/py_stringmatching/v0.3.x/Ratio.html) : \n",
    "\n",
    "You can see one example to understand this furhter. \n",
    "```\n",
    "String1: \"this is a test\"\n",
    "String2: \"this is a test!\"\n",
    "\n",
    "Fuzz Ratio => 97  #\n",
    "\n",
    "Fuzz Partial Ratio => 100  #Since most characters are the same and in a similar sequence, the algorithm calculates the partial ratio as 100 and ignores simple additions (new characters). \n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7b170579a455"
   },
   "source": [
    "First, install the package `fuzzywuzzy` and `python-Levenshtein`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6c55ea0eaed0"
   },
   "outputs": [],
   "source": [
    "!pip install -q python-Levenshtein --upgrade --user\n",
    "!pip install -q fuzzywuzzy --upgrade --user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5f048152519f"
   },
   "source": [
    "Then compute a score to perform fuzzy matching:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "040c1f9a175b"
   },
   "outputs": [],
   "source": [
    "from fuzzywuzzy import fuzz\n",
    "\n",
    "\n",
    "def get_fuzzy_match(df):\n",
    "    return fuzz.partial_ratio(df[\"answer_groundtruth\"], df[\"answer_prediction\"])\n",
    "\n",
    "\n",
    "qa_data_df[\"match_score\"] = qa_data_df.apply(get_fuzzy_match, axis=1)\n",
    "qa_data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "11e266c49860"
   },
   "source": [
    "Now that you have the individual match score (partial), you can take the mean or average of the whole column to get a sense of overall data. \n",
    "Scores closer to 100 mean PaLM 2 can predict closer to ground truth; if the score is towards 50 or 0, it did not perform well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dae6a92a7650"
   },
   "outputs": [],
   "source": [
    "print(\n",
    "    \"the average match score of all predicted answer from PaLM 2 is : \",\n",
    "    qa_data_df[\"match_score\"].mean(),\n",
    "    \" %\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d9e78972cad1"
   },
   "source": [
    "In this case, you get 100% as the mean score, even though some predictions were missing some words. That means you are very close to the ground truth, and some answers are just missing the exact verboseness of the ground truth. "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "question_answering.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "common-cpu.m108",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/base-cpu:m108"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}