tutorials/14_Query_Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O-W2ZQ6CN-gZ",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Query Classifier Tutorial\n",
    "\n",
    "One of the great benefits of using state-of-the-art NLP models like those available in Haystack is that it allows users to state their queries as *plain natural language questions*: rather than trying to come up with just the right set of keywords to find the answer to their question, users can simply ask their question in much the same way that they would ask it of a (very knowledgeable!) person.\n",
    "\n",
    "But just because users *can* ask their questions in \"plain English\" (or \"plain German\", etc.), that doesn't mean they always *will*. For instance, a user might input a few keywords rather than a complete question because they don't understand the pipeline's full capabilities, or because they are so accustomed to keyword search. While a standard Haystack pipeline might handle such queries with reasonable accuracy, for a variety of reasons we still might prefer that our pipeline be sensitive to the type of query it is receiving, so that it behaves differently when a user inputs, say, a collection of keywords instead of a question.\n",
    "\n",
    "For this reason, Haystack comes with built-in capabilities to distinguish between three types of queries: **keyword queries**, **interrogative queries**, and **statement queries**, described below.\n",
    "\n",
    "1. **Keyword queries** can be thought of more or less as lists of words, such as \"Alaska cruises summer\". While the meanings of individual words may matter in a keyword query, the linguistic connections *between* words do not. Hence, in a keyword query the order of words is largely irrelevant: \"Alaska cruises summer\", \"summer Alaska cruises\", and \"summer cruises Alaska\" are functionally the same.\n",
    "\n",
    "2. **Interrogative queries** (or **question queries**) are queries phrased as natural language questions, such as \"Who was the father of Eddard Stark?\". Unlike with keyword queries, word order very much matters here: \"Who was the father of Eddard Stark?\" and \"Who was Eddard Stark the father of?\" are very different questions, despite having exactly the same words. (Note that while we often write questions with question marks, Haystack can find interrogative queries without such a dead giveaway!)\n",
    "\n",
    "3. **Statement queries** are just declarative sentences, such as \"Daenerys loved Jon\". These are like interrogative queries in that word order matters&mdash;again, \"Daenerys loved Jon\" and \"Jon loved Daenerys\" mean very different things&mdash;but they are statements instead of questions.\n",
    "\n",
    "In this tutorial you will learn how to use **query classifiers** to branch your Haystack pipeline based on the type of query it receives. Haystack comes with two out-of-the-box query classification schemas, each of which routes a given query into one of two branches:\n",
    "\n",
    "1. **Keyword vs. Question/Statement** &mdash; routes a query into one of two branches depending on whether it is a full question/statement or a collection of keywords.\n",
    "\n",
    "2. **Question vs. Statement** &mdash; routes a natural language query into one of two branches depending on whether it is a question or a statement.\n",
    "\n",
    "Furthermore, for each classification schema there are two types of nodes capable of performing this classification: a **`TransformersQueryClassifier`** that uses a transformer model, and an **`SklearnQueryClassifier`** that uses a more lightweight model built in `sklearn`.\n",
    "\n",
    "With all of that explanation out of the way, let's dive in!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yaaKv3_ZN-gb",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Prepare the Environment\n",
    "\n",
    "#### Colab: Enable the GPU runtime\n",
    "Make sure you enable the GPU runtime to experience decent speed in this tutorial.  \n",
    "**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/deepset-ai/haystack/main/docs/img/colab_gpu_runtime.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TNlqD5HeN-gc",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next we make sure the latest version of Haystack is installed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "CjA5n5lMN-gd",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Install the latest release of Haystack in your own environment\n",
    "#! pip install farm-haystack\n",
    "\n",
    "# Install the latest main of Haystack (Colab)\n",
    "!pip install --upgrade pip\n",
    "!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]\n",
    "\n",
    "# Install these to allow pipeline visualization\n",
    "!apt install libgraphviz-dev\n",
    "!pip install pygraphviz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Logging\n",
    "\n",
    "We configure how logging messages should be displayed and which log level should be used before importing Haystack.\n",
    "Example log message:\n",
    "INFO - haystack.utils.preprocessing -  Converting data/tutorial1/218_Olenna_Tyrell.txt\n",
    "Default log level in basicConfig is WARNING so the explicit parameter is not necessary but can be changed easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "logging.basicConfig(format=\"%(levelname)s - %(name)s -  %(message)s\", level=logging.WARNING)\n",
    "logging.getLogger(\"haystack\").setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sJcWRK4Hwyx2"
   },
   "source": [
    "### Trying Some Query Classifiers on their Own\n",
    "\n",
    "Before integrating query classifiers into our pipelines, let's test them out on their own and see what they actually do. First we initiate a simple, out-of-the-box **keyword vs. question/statement** `SklearnQueryClassifier`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "XhPMEqBzxA8V"
   },
   "outputs": [],
   "source": [
    "# Here we create the keyword vs question/statement query classifier\n",
    "from haystack.nodes import SklearnQueryClassifier\n",
    "\n",
    "keyword_classifier = SklearnQueryClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1NHjy9aa9FKx"
   },
   "source": [
    "Now let's feed some queries into this query classifier. We'll test with one keyword query, one interrogative query, and one statement query. Note that we don't need to use any punctuation, such as question marks, for the query classifier to make the right decision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ks7qdULR8J13"
   },
   "outputs": [],
   "source": [
    "queries = [\n",
    "    \"Arya Stark father\",  # Keyword Query\n",
    "    \"Who was the father of Arya Stark\",  # Interrogative Query\n",
    "    \"Lord Eddard was the father of Arya Stark\",  # Statement Query\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UbKlyXcNj-nx"
   },
   "source": [
    "Below, you can see what the classifier does with these queries: it correctly determines that  \"Arya Stark father\" is a keyword query and sends it to branch 2. It also correctly classifies both the interrogative query \"Who was the father of Arya Stark\" and the statement query \"Lord Eddard was the father of Arya Stark\" as non-keyword queries, and sends them to branch 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NYROmSHnE4zp"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "k_vs_qs_results = {\"Query\": [], \"Output Branch\": [], \"Class\": []}\n",
    "\n",
    "for query in queries:\n",
    "    result = keyword_classifier.run(query=query)\n",
    "    k_vs_qs_results[\"Query\"].append(query)\n",
    "    k_vs_qs_results[\"Output Branch\"].append(result[1])\n",
    "    k_vs_qs_results[\"Class\"].append(\"Question/Statement\" if result[1] == \"output_1\" else \"Keyword\")\n",
    "\n",
    "pd.DataFrame.from_dict(k_vs_qs_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VyMZzRVHlG5O"
   },
   "source": [
    "Next, we will illustrate a **question vs. statement** `SklearnQueryClassifier`. We define our classifier below. Note that this time we have to explicitly specify the model and vectorizer since the default for a `SklearnQueryClassifier` (and a `TransformersQueryClassifier`) is keyword vs. question/statement classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "l4eH3SSaxZ0O"
   },
   "outputs": [],
   "source": [
    "# Here we create the question vs statement query classifier\n",
    "model_url = (\n",
    "    \"https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier_statements/model.pickle\"\n",
    ")\n",
    "vectorizer_url = (\n",
    "    \"https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier_statements/vectorizer.pickle\"\n",
    ")\n",
    "\n",
    "question_classifier = SklearnQueryClassifier(model_name_or_path=model_url, vectorizer_name_or_path=vectorizer_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zdAY1CUYnTFa"
   },
   "source": [
    "We will test this classifier on the two question/statement queries from the last go-round:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1ZULHEBVmqq2"
   },
   "outputs": [],
   "source": [
    "queries = [\n",
    "    \"Who was the father of Arya Stark\",  # Interrogative Query\n",
    "    \"Lord Eddard was the father of Arya Stark\",  # Statement Query\n",
    "]\n",
    "\n",
    "q_vs_s_results = {\"Query\": [], \"Output Branch\": [], \"Class\": []}\n",
    "\n",
    "for query in queries:\n",
    "    result = question_classifier.run(query=query)\n",
    "    q_vs_s_results[\"Query\"].append(query)\n",
    "    q_vs_s_results[\"Output Branch\"].append(result[1])\n",
    "    q_vs_s_results[\"Class\"].append(\"Question\" if result[1] == \"output_1\" else \"Statement\")\n",
    "\n",
    "pd.DataFrame.from_dict(q_vs_s_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fk2kpvQR6Fa0"
   },
   "source": [
    "And as we see, the question \"Who was the father of Arya Stark\" is sent to branch 1, while the statement \"Lord Eddard was the father of Arya Stark\" is sent to branch 2. This means we can have our pipeline treat statements and questions differently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eEwDIq9KXXke"
   },
   "source": [
    "### Using Query Classifiers in a Pipeline\n",
    "\n",
    "Now let's see how we can use query classifiers in a question-answering (QA) pipeline. We start by initiating Elasticsearch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fCLtLItU5aWl"
   },
   "outputs": [],
   "source": [
    "# In Colab / No Docker environments: Start Elasticsearch from source\n",
    "! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q\n",
    "! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz\n",
    "! chown -R daemon:daemon elasticsearch-7.9.2\n",
    "\n",
    "import os\n",
    "from subprocess import Popen, PIPE, STDOUT\n",
    "\n",
    "es_server = Popen(\n",
    "    [\"elasticsearch-7.9.2/bin/elasticsearch\"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1)  # as daemon\n",
    ")\n",
    "# wait until ES has started\n",
    "! sleep 30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vm9gqTioN-gf",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next we fetch some data&mdash;for our example we'll use pages from the Game of Thrones wiki&mdash;and index it in our `DocumentStore`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "Ig7dgfdHN-gg",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from haystack.utils import (\n",
    "    print_answers,\n",
    "    print_documents,\n",
    "    fetch_archive_from_http,\n",
    "    convert_files_to_docs,\n",
    "    clean_wiki_text,\n",
    "    launch_es,\n",
    ")\n",
    "from haystack.pipelines import Pipeline\n",
    "from haystack.document_stores import ElasticsearchDocumentStore\n",
    "from haystack.nodes import BM25Retriever, EmbeddingRetriever, FARMReader, TransformersQueryClassifier\n",
    "\n",
    "# Download and prepare data - 517 Wikipedia articles for Game of Thrones\n",
    "doc_dir = \"data/tutorial14\"\n",
    "s3_url = \"https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt14.zip\"\n",
    "fetch_archive_from_http(url=s3_url, output_dir=doc_dir)\n",
    "\n",
    "# convert files to dicts containing documents that can be indexed to our datastore\n",
    "got_docs = convert_files_to_docs(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)\n",
    "\n",
    "# Initialize DocumentStore and index documents\n",
    "# launch_es() # Uncomment this line for local Elasticsearch\n",
    "document_store = ElasticsearchDocumentStore()\n",
    "document_store.delete_documents()\n",
    "document_store.write_documents(got_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CbAgZ2MZn2qm"
   },
   "source": [
    "#### Pipelines with Keyword vs. Question/Statement Classification\n",
    "\n",
    "Our first illustration will be a simple retriever-reader QA pipeline, but the choice of which retriever we use will depend on the type of query received: **keyword** queries will use a sparse **`BM25Retriever`**, while **question/statement** queries will use the more accurate but also more computationally expensive **`EmbeddingRetriever`**.\n",
    "\n",
    "We start by initializing our retrievers and reader:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "m7zOPYQ-Ylep"
   },
   "outputs": [],
   "source": [
    "# Initialize sparse retriever for keyword queries\n",
    "bm25_retriever = BM25Retriever(document_store=document_store)\n",
    "\n",
    "# Initialize dense retriever for question/statement queries\n",
    "embedding_retriever = EmbeddingRetriever(\n",
    "    document_store=document_store, embedding_model=\"sentence-transformers/multi-qa-mpnet-base-dot-v1\"\n",
    ")\n",
    "document_store.update_embeddings(embedding_retriever, update_existing_embeddings=False)\n",
    "\n",
    "reader = FARMReader(model_name_or_path=\"deepset/roberta-base-squad2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K4wZ3xkQCHjY"
   },
   "source": [
    "Now we define our pipeline. As promised, the question/statement branch `output_1` from the query classifier is fed into an `EmbeddingRetriever`, while the keyword branch `output_2` from the same classifier is fed into a `BM25Retriever`. Both of these retrievers are then fed into our reader. Our pipeline can thus be thought of as having something of a diamond shape: all queries are sent into the classifier, which splits those queries into two different retrievers, and those retrievers feed their outputs to the same reader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "Sz-oZ5eJN-gl",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Here we build the pipeline\n",
    "sklearn_keyword_classifier = Pipeline()\n",
    "sklearn_keyword_classifier.add_node(component=SklearnQueryClassifier(), name=\"QueryClassifier\", inputs=[\"Query\"])\n",
    "sklearn_keyword_classifier.add_node(\n",
    "    component=embedding_retriever, name=\"EmbeddingRetriever\", inputs=[\"QueryClassifier.output_1\"]\n",
    ")\n",
    "sklearn_keyword_classifier.add_node(component=bm25_retriever, name=\"BM25Retriever\", inputs=[\"QueryClassifier.output_2\"])\n",
    "sklearn_keyword_classifier.add_node(component=reader, name=\"QAReader\", inputs=[\"BM25Retriever\", \"EmbeddingRetriever\"])\n",
    "\n",
    "# Visualization of the pipeline\n",
    "sklearn_keyword_classifier.draw(\"sklearn_keyword_classifier.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "imqRRCGTwQav"
   },
   "source": [
    "Below, we can see how this choice affects the branching structure: the keyword query \"arya stark father\" and the question query \"Who is the father of Arya Stark?\" generate noticeably different results, a distinction that is likely due to the use of different retrievers for keyword vs. question/statement queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "fP6Cpcb-o0HK"
   },
   "outputs": [],
   "source": [
    "# Useful for framing headers\n",
    "equal_line = \"=\" * 30\n",
    "\n",
    "# Run only the dense retriever on the full sentence query\n",
    "res_1 = sklearn_keyword_classifier.run(query=\"Who is the father of Arya Stark?\")\n",
    "print(f\"\\n\\n{equal_line}\\nQUESTION QUERY RESULTS\\n{equal_line}\")\n",
    "print_answers(res_1, details=\"minimum\")\n",
    "print(\"\\n\\n\")\n",
    "\n",
    "# Run only the sparse retriever on a keyword based query\n",
    "res_2 = sklearn_keyword_classifier.run(query=\"arya stark father\")\n",
    "print(f\"\\n\\n{equal_line}\\nKEYWORD QUERY RESULTS\\n{equal_line}\")\n",
    "print_answers(res_2, details=\"minimum\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dQ5YMyd4CQPC"
   },
   "source": [
    "The above example uses an `SklearnQueryClassifier`, but of course we can do precisely the same thing with a `TransformersQueryClassifier`. This is illustrated below, where we have constructed the same diamond-shaped pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yuddZL3FCPeq"
   },
   "outputs": [],
   "source": [
    "# Here we build the pipeline\n",
    "transformer_keyword_classifier = Pipeline()\n",
    "transformer_keyword_classifier.add_node(\n",
    "    component=TransformersQueryClassifier(), name=\"QueryClassifier\", inputs=[\"Query\"]\n",
    ")\n",
    "transformer_keyword_classifier.add_node(\n",
    "    component=embedding_retriever, name=\"EmbeddingRetriever\", inputs=[\"QueryClassifier.output_1\"]\n",
    ")\n",
    "transformer_keyword_classifier.add_node(\n",
    "    component=bm25_retriever, name=\"BM25Retriever\", inputs=[\"QueryClassifier.output_2\"]\n",
    ")\n",
    "transformer_keyword_classifier.add_node(\n",
    "    component=reader, name=\"QAReader\", inputs=[\"BM25Retriever\", \"EmbeddingRetriever\"]\n",
    ")\n",
    "\n",
    "\n",
    "# Useful for framing headers\n",
    "equal_line = \"=\" * 30\n",
    "\n",
    "# Run only the dense retriever on the full sentence query\n",
    "res_1 = transformer_keyword_classifier.run(query=\"Who is the father of Arya Stark?\")\n",
    "print(f\"\\n\\n{equal_line}\\nQUESTION QUERY RESULTS\\n{equal_line}\")\n",
    "print_answers(res_1, details=\"minimum\")\n",
    "print(\"\\n\\n\")\n",
    "\n",
    "# Run only the sparse retriever on a keyword based query\n",
    "res_2 = transformer_keyword_classifier.run(query=\"arya stark father\")\n",
    "print(f\"\\n\\n{equal_line}\\nKEYWORD QUERY RESULTS\\n{equal_line}\")\n",
    "print_answers(res_2, details=\"minimum\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zLwdVwMXDcoS"
   },
   "source": [
    "#### Pipeline with Question vs. Statement Classification\n",
    "\n",
    "Above we saw a potential use for keyword vs. question/statement classification: we might choose to use a less resource-intensive retriever for keyword queries than for question/statement queries. But what about question vs. statement classification?\n",
    "\n",
    "To illustrate one potential use for question vs. statement classification, we will build a pipeline that looks as follows:\n",
    "\n",
    "1. The pipeline will start with a retriever that **every query** will go through.\n",
    "2. The pipeline will end with a reader that **only question queries** will go through.\n",
    "\n",
    "In other words, our pipeline will be a **retriever-only pipeline for statement queries**&mdash;given the statement \"Arya Stark was the daughter of a Lord\", all we will get back are the most relevant documents&mdash;but it will be a **retriever-reader pipeline for question queries**.\n",
    "\n",
    "To make things more concrete, our pipeline will start with a retriever, which is then fed into a `TransformersQueryClassifier` that is set to do question vs. statement classification. Note that this means we need to explicitly choose the model, since as mentioned previously a default `TransformersQueryClassifier` performs keyword vs. question/statement classification. The classifier's first branch, which handles question queries, will then be sent to the reader, while the second branch will not be connected to any other nodes. As a result, the last node of the pipeline depends on the type of query: questions go all the way through the reader, while statements only go through the retriever. This pipeline is illustrated below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "BIisEJrzDr-9"
   },
   "outputs": [],
   "source": [
    "# Here we build the pipeline\n",
    "transformer_question_classifier = Pipeline()\n",
    "transformer_question_classifier.add_node(component=embedding_retriever, name=\"EmbeddingRetriever\", inputs=[\"Query\"])\n",
    "transformer_question_classifier.add_node(\n",
    "    component=TransformersQueryClassifier(model_name_or_path=\"shahrukhx01/question-vs-statement-classifier\"),\n",
    "    name=\"QueryClassifier\",\n",
    "    inputs=[\"EmbeddingRetriever\"],\n",
    ")\n",
    "transformer_question_classifier.add_node(component=reader, name=\"QAReader\", inputs=[\"QueryClassifier.output_1\"])\n",
    "\n",
    "# Visualization of the pipeline\n",
    "transformer_question_classifier.draw(\"transformer_question_classifier.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QU1B6JQEDrol"
   },
   "source": [
    "And here are the results of this pipeline: with a question query like \"Who is the father of Arya Stark?\", we obtain answers from a reader, and with a statement query like \"Arya Stark was the daughter of a Lord\", we just obtain documents from a retriever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HIjgs5k7C6CN"
   },
   "outputs": [],
   "source": [
    "# Useful for framing headers\n",
    "equal_line = \"=\" * 30\n",
    "\n",
    "# Run the retriever + reader on the question query\n",
    "res_1 = transformer_question_classifier.run(query=\"Who is the father of Arya Stark?\")\n",
    "print(f\"\\n\\n{equal_line}\\nQUESTION QUERY RESULTS\\n{equal_line}\")\n",
    "print_answers(res_1, details=\"minimum\")\n",
    "print(\"\\n\\n\")\n",
    "\n",
    "# Run only the retriever on the statement query\n",
    "res_2 = transformer_question_classifier.run(query=\"Arya Stark was the daughter of a Lord.\")\n",
    "print(f\"\\n\\n{equal_line}\\nSTATEMENT QUERY RESULTS\\n{equal_line}\")\n",
    "print_documents(res_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other use cases for Query Classifiers: custom classification models and zero-shot classification.\n",
    "\n",
    "`TransformersQueryClassifier` is very flexible and also supports other options for classifying queries.\n",
    "For example, we may be interested in detecting the sentiment or classifying the topics.  We can do this by loading a custom classification model from the Hugging Face Hub or by using zero-shot classification.\n",
    "\n",
    "#### Custom classification model vs zero-shot classification\n",
    "- Rraditional text classification models are trained to predict one of a few \"hard-coded\" classes and require a dedicated training dataset. In the Hugging Face Hub, you can find many pre-trained models, maybe even related to your domain of interest.\n",
    "- Zero-shot classification is very versatile: by choosing a suitable base transformer, you can classify the text without any training dataset. You just have to provide the candidate categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using custom classification models\n",
    "We can use a public model, available in the Hugging Face Hub. For example, if we want to classify the sentiment of the queries, we can choose an appropriate model, such as https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment.\n",
    "\n",
    "*In this case, the `labels` parameter must contain a list with the exact model labels.\n",
    "The first label we provide corresponds to output_1, the second label to output_2, and so on.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.nodes import TransformersQueryClassifier\n",
    "\n",
    "# Remember to compile a list with the exact model labels\n",
    "# The first label you provide corresponds to output_1, the second label to output_2, and so on.\n",
    "labels = [\"LABEL_0\", \"LABEL_1\", \"LABEL_2\"]\n",
    "\n",
    "sentiment_query_classifier = TransformersQueryClassifier(\n",
    "    model_name_or_path=\"cardiffnlp/twitter-roberta-base-sentiment\",\n",
    "    use_gpu=True,\n",
    "    task=\"text-classification\",\n",
    "    labels=labels,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "queries = [\n",
    "    \"What's the answer?\",  # neutral query\n",
    "    \"Would you be so lovely to tell me the answer?\",  # positive query\n",
    "    \"Can you give me the damn right answer for once??\",  # negative query\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "sent_results = {\"Query\": [], \"Output Branch\": [], \"Class\": []}\n",
    "\n",
    "for query in queries:\n",
    "    result = sentiment_query_classifier.run(query=query)\n",
    "    sent_results[\"Query\"].append(query)\n",
    "    sent_results[\"Output Branch\"].append(result[1])\n",
    "    if result[1] == \"output_1\":\n",
    "        sent_results[\"Class\"].append(\"negative\")\n",
    "    elif result[1] == \"output_2\":\n",
    "        sent_results[\"Class\"].append(\"neutral\")\n",
    "    elif result[1] == \"output_3\":\n",
    "        sent_results[\"Class\"].append(\"positive\")\n",
    "\n",
    "pd.DataFrame.from_dict(sent_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using zero-shot classification\n",
    "You can also perform zero-shot classification by providing a suitable base transformer model and **choosing** the classes the model should predict.\n",
    "For example, we may be interested in whether the user query is related to music or cinema.\n",
    "\n",
    "*In this case, the `labels` parameter is a list containing the candidate classes.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In zero-shot-classification, you can choose the labels\n",
    "labels = [\"music\", \"cinema\"]\n",
    "\n",
    "query_classifier = TransformersQueryClassifier(\n",
    "    model_name_or_path=\"typeform/distilbert-base-uncased-mnli\",\n",
    "    use_gpu=True,\n",
    "    task=\"zero-shot-classification\",\n",
    "    labels=labels,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "queries = [\n",
    "    \"In which films does John Travolta appear?\",  # query about cinema\n",
    "    \"What is the Rolling Stones first album?\",  # query about music\n",
    "    \"Who was Sergio Leone?\",  # query about cinema\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "query_classification_results = {\"Query\": [], \"Output Branch\": [], \"Class\": []}\n",
    "\n",
    "for query in queries:\n",
    "    result = query_classifier.run(query=query)\n",
    "    query_classification_results[\"Query\"].append(query)\n",
    "    query_classification_results[\"Output Branch\"].append(result[1])\n",
    "    query_classification_results[\"Class\"].append(\"music\" if result[1] == \"output_1\" else \"cinema\")\n",
    "\n",
    "pd.DataFrame.from_dict(query_classification_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_wS8NzRoRh_G"
   },
   "source": [
    "## About us\n",
    "\n",
    "This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany\n",
    "\n",
    "We bring NLP to the industry via open source!  \n",
    "Our focus: Industry specific language models & large scale QA systems.\n",
    "  \n",
    "Some of our other work: \n",
    "- [German BERT](https://deepset.ai/german-bert)\n",
    "- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)\n",
    "- [FARM](https://github.com/deepset-ai/FARM)\n",
    "\n",
    "Get in touch:\n",
    "[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Discord](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)\n",
    "\n",
    "By the way: [we're hiring!](https://www.deepset.ai/jobs) "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Tutorial14_Query_Classifier.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.7.11 ('haystack-dev')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  },
  "vscode": {
   "interpreter": {
    "hash": "a1c4180befe5334d9af26d84758dc08f43161c3b98a4eb4d4a43d7491d015a65"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}