tutorials/Tutorial16_Document_Classifier_at_Index_Time.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extending your Metadata using DocumentClassifiers at Index Time\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial16_Document_Classifier_at_Index_Time.ipynb)\n",
    "\n",
    "With DocumentClassifier it's possible to automatically enrich your documents with categories, sentiments, topics or whatever metadata you like. This metadata could be used for efficient filtering or further processing. Say you have some categories your users typically filter on. If the documents are tagged manually with these categories, you could automate this process by training a model. Or you can leverage the full power and flexibility of zero shot classification. All you need to do is pass your categories to the classifier, no labels required. This tutorial shows how to integrate it in your indexing pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DocumentClassifier adds the classification result (label and score) to Document's meta property.\n",
    "Hence, we can use it to classify documents at index time. \\\n",
    "The result can be accessed at query time: for example by applying a filter for \"classification.label\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This tutorial will show you how to integrate a classification model into your preprocessing steps and how you can filter for this additional metadata at query time. In the last section we show how to put it all together and create an indexing pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Install the latest release of Haystack in your own environment\n",
    "#! pip install farm-haystack\n",
    "\n",
    "# Install the latest master of Haystack\n",
    "!pip install --upgrade pip\n",
    "!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab,ocr]\n",
    "\n",
    "!wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.04.tar.gz\n",
    "!tar -xvf xpdf-tools-linux-4.04.tar.gz && sudo cp xpdf-tools-linux-4.04/bin64/pdftotext /usr/local/bin\n",
    "\n",
    "# Install  pygraphviz\n",
    "!apt install libgraphviz-dev\n",
    "!pip install pygraphviz"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Logging\n",
    "\n",
    "We configure how logging messages should be displayed and which log level should be used before importing Haystack.\n",
    "Example log message:\n",
    "INFO - haystack.utils.preprocessing -  Converting data/tutorial1/218_Olenna_Tyrell.txt\n",
    "Default log level in basicConfig is WARNING so the explicit parameter is not necessary but can be changed easily:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "logging.basicConfig(format=\"%(levelname)s - %(name)s -  %(message)s\", level=logging.WARNING)\n",
    "logging.getLogger(\"haystack\").setLevel(logging.INFO)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Here are the imports we need\n",
    "from haystack.document_stores.elasticsearch import ElasticsearchDocumentStore\n",
    "from haystack.nodes import PreProcessor, TransformersDocumentClassifier, FARMReader, BM25Retriever\n",
    "from haystack.schema import Document\n",
    "from haystack.utils import convert_files_to_docs, fetch_archive_from_http, print_answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This fetches some sample files to work with\n",
    "\n",
    "doc_dir = \"data/tutorial16\"\n",
    "s3_url = \"https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/preprocessing_tutorial16.zip\"\n",
    "fetch_archive_from_http(url=s3_url, output_dir=doc_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Read and preprocess documents\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "pdftotext version 0.86.1\n",
      "Copyright 2005-2020 The Poppler Developers - http:https://poppler.freedesktop.org\n",
      "Copyright 1996-2011 Glyph & Cog, LLC\n",
      "100%|██████████| 3/3 [00:00<00:00, 372.17docs/s]\n"
     ]
    }
   ],
   "source": [
    "# note that you can also use the document classifier before applying the PreProcessor, e.g. before splitting your documents\n",
    "\n",
    "all_docs = convert_files_to_docs(dir_path=doc_dir)\n",
    "preprocessor_sliding_window = PreProcessor(split_overlap=3, split_length=10, split_respect_sentence_boundary=False)\n",
    "docs_sliding_window = preprocessor_sliding_window.process(all_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply DocumentClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can enrich the document metadata at index time using any transformers document classifier model. While traditional classification models are trained to predict one of a few \"hard-coded\" classes and required a dedicated training dataset, zero-shot classification is super flexible and you can easily switch the classes the model should predict on the fly. Just supply them via the labels param.\n",
    "Here we use a zero shot model that is supposed to classify our documents in 'music', 'natural language processing' and 'history'. Feel free to change them for whatever you like to classify. \\\n",
    "These classes can later on be accessed at query time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_classifier = TransformersDocumentClassifier(\n",
    "    model_name_or_path=\"cross-encoder/nli-distilroberta-base\",\n",
    "    task=\"zero-shot-classification\",\n",
    "    labels=[\"music\", \"natural language processing\", \"history\"],\n",
    "    batch_size=16,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also use any other transformers model besides zero shot classification\n",
    "\n",
    "# doc_classifier_model = 'bhadresh-savani/distilbert-base-uncased-emotion'\n",
    "# doc_classifier = TransformersDocumentClassifier(model_name_or_path=doc_classifier_model, batch_size=16, use_gpu=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we could also specifiy a different field we want to run the classification on\n",
    "\n",
    "# doc_classifier = TransformersDocumentClassifier(model_name_or_path=\"cross-encoder/nli-distilroberta-base\",\n",
    "#    task=\"zero-shot-classification\",\n",
    "#    labels=[\"music\", \"natural language processing\", \"history\"],\n",
    "#    batch_size=16, use_gpu=-1,\n",
    "#    classification_field=\"description\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# classify using gpu, batch_size makes sure we do not run out of memory\n",
    "classified_docs = doc_classifier.predict(docs_sliding_window)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'content': 'Heavy metal\\n\\nHeavy metal (or simply metal) is a genre of', 'content_type': 'text', 'score': None, 'meta': {'name': 'heavy_metal.docx', '_split_id': 0, 'classification': {'sequence': 'Heavy metal\\n\\nHeavy metal (or simply metal) is a genre of', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.8191022872924805, 0.11593689769506454, 0.06496082246303558], 'label': 'music'}}, 'embedding': None, 'id': '9903d23737f3d05a9d9ee170703dc245'}\n"
     ]
    }
   ],
   "source": [
    "# let's see how it looks: there should be a classification result in the meta entry containing labels and scores.\n",
    "print(classified_docs[0].to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In Colab / No Docker environments: Start Elasticsearch from source\n",
    "! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q\n",
    "! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz\n",
    "! chown -R daemon:daemon elasticsearch-7.9.2\n",
    "\n",
    "import os\n",
    "from subprocess import Popen, PIPE, STDOUT\n",
    "\n",
    "es_server = Popen(\n",
    "    [\"elasticsearch-7.9.2/bin/elasticsearch\"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1)  # as daemon\n",
    ")\n",
    "# wait until ES has started\n",
    "! sleep 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to Elasticsearch\n",
    "document_store = ElasticsearchDocumentStore(host=\"localhost\", username=\"\", password=\"\", index=\"document\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "DEPRECATION WARNINGS: \n",
      "                1. delete_all_documents() method is deprecated, please use delete_documents method\n",
      "                For more details, please refer to the issue: https://github.com/deepset-ai/haystack/issues/1045\n",
      "                \n"
     ]
    }
   ],
   "source": [
    "# Now, let's write the docs to our DB.\n",
    "document_store.delete_all_documents()\n",
    "document_store.write_documents(classified_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "document 9903d23737f3d05a9d9ee170703dc245 with content \n",
      "\n",
      "Heavy metal\n",
      "\n",
      "Heavy metal (or simply metal) is a genre of\n",
      "\n",
      "has label music\n"
     ]
    }
   ],
   "source": [
    "# check if indexed docs contain classification results\n",
    "test_doc = document_store.get_all_documents()[0]\n",
    "print(\n",
    "    f'document {test_doc.id} with content \\n\\n{test_doc.content}\\n\\nhas label {test_doc.meta[\"classification\"][\"label\"]}'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All we have to do to filter for one of our classes is to set a filter on \"classification.label\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize QA-Pipeline\n",
    "from haystack.pipelines import ExtractiveQAPipeline\n",
    "\n",
    "retriever = BM25Retriever(document_store=document_store)\n",
    "reader = FARMReader(model_name_or_path=\"deepset/roberta-base-squad2\", use_gpu=True)\n",
    "pipe = ExtractiveQAPipeline(reader, retriever)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Inferencing Samples:   0%|          | 0/1 [00:00<?, ? Batches/s]/home/tstad/git/haystack/haystack/modeling/model/prediction_head.py:455: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "  start_indices = flat_sorted_indices // max_seq_len\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.97 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.45 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.46 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.46 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.44 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.43 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.48 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.47 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.48 Batches/s]\n",
      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.46 Batches/s]\n"
     ]
    }
   ],
   "source": [
    "## Voilà! Ask a question while filtering for \"music\"-only documents\n",
    "prediction = pipe.run(\n",
    "    query=\"What is heavy metal?\",\n",
    "    params={\"Retriever\": {\"top_k\": 10, \"filters\": {\"classification.label\": [\"music\"]}}, \"Reader\": {\"top_k\": 5}},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{   'answers': [   Answer(answer='thick, massive sound', type='extractive', score=0.5388454645872116, context=',[6] heavy metal bands developed a thick, massive sound, characterized', offsets_in_document=[Span(start=35, end=55)], offsets_in_context=[Span(start=35, end=55)], document_id='b69a8816c2c8d782dceb412b80a4bf6e', meta={'_split_id': 5, 'classification': {'sequence': ',[6] heavy metal bands developed a thick, massive sound, characterized', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.926899254322052, 0.0447617806494236, 0.0283389650285244], 'label': 'music'}, 'name': 'heavy_metal.docx'}),\n",
      "                   Answer(answer='a genre', type='extractive', score=0.399858795106411, context='Heavy metal\\n\\nHeavy metal (or simply metal) is a genre of', offsets_in_document=[Span(start=46, end=53)], offsets_in_context=[Span(start=46, end=53)], document_id='9903d23737f3d05a9d9ee170703dc245', meta={'_split_id': 0, 'classification': {'sequence': 'Heavy metal\\n\\nHeavy metal (or simply metal) is a genre of', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.8191022872924805, 0.11593689769506454, 0.06496082246303558], 'label': 'music'}, 'name': 'heavy_metal.docx'}),\n",
      "                   Answer(answer='more accessible forms', type='extractive', score=0.09895830601453781, context='Several American bands modified heavy metal into more accessible forms', offsets_in_document=[Span(start=49, end=70)], offsets_in_context=[Span(start=49, end=70)], document_id='5a9cefc4732e4b2f97529a79231345ec', meta={'_split_id': 13, 'classification': {'sequence': 'Several American bands modified heavy metal into more accessible forms', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.9277411699295044, 0.036878395825624466, 0.035380441695451736], 'label': 'music'}, 'name': 'heavy_metal.docx'}),\n",
      "                   Answer(answer='roots', type='extractive', score=0.017235582694411278, context='States.[5] With roots in , , and ,[6] heavy metal', offsets_in_document=[Span(start=16, end=21)], offsets_in_context=[Span(start=16, end=21)], document_id='7aa5f844f509c36ee6cf6a8d605ca452', meta={'_split_id': 4, 'classification': {'sequence': 'States.[5] With roots in , , and ,[6] heavy metal', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.661548376083374, 0.20618689060211182, 0.13226467370986938], 'label': 'music'}, 'name': 'heavy_metal.docx'}),\n",
      "                   Answer(answer='heavy metal fans', type='extractive', score=0.014702926389873028, context='decade, heavy metal fans became known as \"\" or \"\".\\nDuring', offsets_in_document=[Span(start=8, end=24)], offsets_in_context=[Span(start=8, end=24)], document_id='ac5cc61f3e646d87d6549a7bb80b8b4a', meta={'_split_id': 26, 'classification': {'sequence': 'decade, heavy metal fans became known as \"\" or \"\".\\nDuring', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.5980438590049744, 0.32206231355667114, 0.0798937976360321], 'label': 'music'}, 'name': 'heavy_metal.docx'})],\n",
      "    'documents': [   {'content': 'Heavy metal\\n\\nHeavy metal (or simply metal) is a genre of', 'content_type': 'text', 'score': 0.9135275466034989, 'meta': {'_split_id': 0, 'classification': {'sequence': 'Heavy metal\\n\\nHeavy metal (or simply metal) is a genre of', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.8191022872924805, 0.11593689769506454, 0.06496082246303558], 'label': 'music'}, 'name': 'heavy_metal.docx'}, 'embedding': None, 'id': '9903d23737f3d05a9d9ee170703dc245'},\n",
      "                     {'content': 'States.[5] With roots in , , and ,[6] heavy metal', 'content_type': 'text', 'score': 0.8256961596331841, 'meta': {'_split_id': 4, 'classification': {'sequence': 'States.[5] With roots in , , and ,[6] heavy metal', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.661548376083374, 0.20618689060211182, 0.13226467370986938], 'label': 'music'}, 'name': 'heavy_metal.docx'}, 'embedding': None, 'id': '7aa5f844f509c36ee6cf6a8d605ca452'},\n",
      "                     {'content': 'decade, heavy metal fans became known as \"\" or \"\".\\nDuring', 'content_type': 'text', 'score': 0.8256961596331841, 'meta': {'_split_id': 26, 'classification': {'sequence': 'decade, heavy metal fans became known as \"\" or \"\".\\nDuring', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.5980438590049744, 0.32206231355667114, 0.0798937976360321], 'label': 'music'}, 'name': 'heavy_metal.docx'}, 'embedding': None, 'id': 'ac5cc61f3e646d87d6549a7bb80b8b4a'},\n",
      "                     {'content': ',[6] heavy metal bands developed a thick, massive sound, characterized', 'content_type': 'text', 'score': 0.8172186978337235, 'meta': {'_split_id': 5, 'classification': {'sequence': ',[6] heavy metal bands developed a thick, massive sound, characterized', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.926899254322052, 0.0447617806494236, 0.0283389650285244], 'label': 'music'}, 'name': 'heavy_metal.docx'}, 'embedding': None, 'id': 'b69a8816c2c8d782dceb412b80a4bf6e'},\n",
      "                     {'content': 'Several American bands modified heavy metal into more accessible forms', 'content_type': 'text', 'score': 0.8172186978337235, 'meta': {'_split_id': 13, 'classification': {'sequence': 'Several American bands modified heavy metal into more accessible forms', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.9277411699295044, 0.036878395825624466, 0.035380441695451736], 'label': 'music'}, 'name': 'heavy_metal.docx'}, 'embedding': None, 'id': '5a9cefc4732e4b2f97529a79231345ec'},\n",
      "                     {'content': 'similar vein. By the end of the decade, heavy metal', 'content_type': 'text', 'score': 0.8172186978337235, 'meta': {'_split_id': 25, 'classification': {'sequence': 'similar vein. By the end of the decade, heavy metal', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.613710343837738, 0.2949920892715454, 0.09129763394594193], 'label': 'music'}, 'name': 'heavy_metal.docx'}, 'embedding': None, 'id': 'f99c648859137f46860d2e14a7524012'},\n",
      "                     {'content': '> snull +  , where the threshold  is', 'content_type': 'text', 'score': 0.5794093707835908, 'meta': {'_split_id': 513, 'classification': {'sequence': '> snull +  , where the threshold  is', 'labels': ['music', 'natural language processing', 'history'], 'scores': [0.7594349384307861, 0.1394801139831543, 0.10108496993780136], 'label': 'music'}, 'name': 'bert.pdf'}, 'embedding': None, 'id': 'dea394d0bc270f451ea72caeb31f8fe0'},\n",
      "                     {'content': 'They are sampled such that the combined length is ', 'content_type': 'text', 'score': 0.5673102196284736, 'meta': {'_split_id': 964, 'classification': {'sequence': 'They are sampled such that the combined length is ', 'labels': ['music', 'natural language processing', 'history'], 'scores': [0.6959176659584045, 0.20470784604549408, 0.0993744507431984], 'label': 'music'}, 'name': 'bert.pdf'}, 'embedding': None, 'id': '3792202c1568920f7353682800f6d3f5'},\n",
      "                     {'content': 'Classics or classical studies is the study of classical antiquity,', 'content_type': 'text', 'score': 0.564837188549428, 'meta': {'_split_id': 0, 'classification': {'sequence': 'Classics or classical studies is the study of classical antiquity,', 'labels': ['music', 'natural language processing', 'history'], 'scores': [0.3462620675563812, 0.33706134557724, 0.3166765868663788], 'label': 'music'}, 'name': 'classics.txt'}, 'embedding': None, 'id': '5f06721d4e5ddd207e8de318274a89b6'},\n",
      "                     {'content': 'Orders: Doric, Ionic, and Corinthian. The Parthenon is still the', 'content_type': 'text', 'score': 0.564837188549428, 'meta': {'_split_id': 272, 'classification': {'sequence': 'Orders: Doric, Ionic, and Corinthian. The Parthenon is still the', 'labels': ['music', 'history', 'natural language processing'], 'scores': [0.8087852001190186, 0.09828957170248032, 0.09292517602443695], 'label': 'music'}, 'name': 'classics.txt'}, 'embedding': None, 'id': 'eb79ba3545ad1c1fa01a32f0b70a7455'}],\n",
      "    'no_ans_gap': 3.725033760070801,\n",
      "    'node_id': 'Reader',\n",
      "    'params': {   'Reader': {'top_k': 5},\n",
      "                  'Retriever': {   'filters': {   'classification.label': [   'music']},\n",
      "                                   'top_k': 10}},\n",
      "    'query': 'What is heavy metal?',\n",
      "    'root_node': 'Query'}\n"
     ]
    }
   ],
   "source": [
    "print_answers(prediction, details=\"high\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping it up in an indexing pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from haystack.pipelines import Pipeline\n",
    "from haystack.nodes import TextConverter, PreProcessor, FileTypeClassifier, PDFToTextConverter, DocxToTextConverter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_type_classifier = FileTypeClassifier()\n",
    "text_converter = TextConverter()\n",
    "pdf_converter = PDFToTextConverter()\n",
    "docx_converter = DocxToTextConverter()\n",
    "\n",
    "indexing_pipeline_with_classification = Pipeline()\n",
    "indexing_pipeline_with_classification.add_node(\n",
    "    component=file_type_classifier, name=\"FileTypeClassifier\", inputs=[\"File\"]\n",
    ")\n",
    "indexing_pipeline_with_classification.add_node(\n",
    "    component=text_converter, name=\"TextConverter\", inputs=[\"FileTypeClassifier.output_1\"]\n",
    ")\n",
    "indexing_pipeline_with_classification.add_node(\n",
    "    component=pdf_converter, name=\"PdfConverter\", inputs=[\"FileTypeClassifier.output_2\"]\n",
    ")\n",
    "indexing_pipeline_with_classification.add_node(\n",
    "    component=docx_converter, name=\"DocxConverter\", inputs=[\"FileTypeClassifier.output_4\"]\n",
    ")\n",
    "indexing_pipeline_with_classification.add_node(\n",
    "    component=preprocessor_sliding_window,\n",
    "    name=\"Preprocessor\",\n",
    "    inputs=[\"TextConverter\", \"PdfConverter\", \"DocxConverter\"],\n",
    ")\n",
    "indexing_pipeline_with_classification.add_node(\n",
    "    component=doc_classifier, name=\"DocumentClassifier\", inputs=[\"Preprocessor\"]\n",
    ")\n",
    "indexing_pipeline_with_classification.add_node(\n",
    "    component=document_store, name=\"DocumentStore\", inputs=[\"DocumentClassifier\"]\n",
    ")\n",
    "indexing_pipeline_with_classification.draw(\"index_time_document_classifier.png\")\n",
    "\n",
    "document_store.delete_documents()\n",
    "txt_files = [f for f in Path(doc_dir).iterdir() if f.suffix == \".txt\"]\n",
    "pdf_files = [f for f in Path(doc_dir).iterdir() if f.suffix == \".pdf\"]\n",
    "docx_files = [f for f in Path(doc_dir).iterdir() if f.suffix == \".docx\"]\n",
    "indexing_pipeline_with_classification.run(file_paths=txt_files)\n",
    "indexing_pipeline_with_classification.run(file_paths=pdf_files)\n",
    "indexing_pipeline_with_classification.run(file_paths=docx_files)\n",
    "\n",
    "document_store.get_all_documents()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can store this pipeline and use it from the REST-API\n",
    "indexing_pipeline_with_classification.save_to_yaml(\"indexing_pipeline_with_classification.yaml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## About us\n",
    "\n",
    "This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany\n",
    "\n",
    "We bring NLP to the industry via open source!  \n",
    "Our focus: Industry specific language models & large scale QA systems.  \n",
    "  \n",
    "Some of our other work: \n",
    "- [German BERT](https://deepset.ai/german-bert)\n",
    "- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)\n",
    "- [FARM](https://github.com/deepset-ai/FARM)\n",
    "\n",
    "Get in touch:\n",
    "[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)\n",
    "\n",
    "By the way: [we're hiring!](https://www.deepset.ai/jobs)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}