tutorials/20_Using_Haystack_with_REST_API.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Using Haystack with REST API\n",
    "\n",
    "- **Level**: Advanced\n",
    "- **Time to complete**: 30 minutes\n",
    "- **Prerequisites**: Basic understanding of Docker and basic knowledge of Haystack pipelines. \n",
    "- **Nodes Used**: `ElasticsearchDocumentStore`, `EmbeddingRetriever`\n",
    "- **Goal**: After you complete this tutorial, you will have learned how to interact with Haystack through REST API."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "Learn how you can interact with Haystack through REST API. This tutorial introduces you to all the concepts needed to build an end-to-end document search application.  \n",
    "\n",
    "With Haystack, you can apply the latest NLP technology to your own data and create production-ready applications. Building an end-to-end NLP application requires the combination of multiple concepts:\n",
    "* **DocumentStore** is the component in Haystack responsible for loading and storing text data in the form of [Documents](https://docs.haystack.deepset.ai/docs/documents_answers_labels#document). In this tutorial, the DocumentStore uses Elasticsearch behind the scene.\n",
    "* **Haystack pipelines** convert files into Documents, index them to the DocumentStore, and run NLP tasks such as question answering and document search.\n",
    "* **REST API**, as a concept, makes it possible for applications to interact with each other by handling their queries and returning responses. There is `rest_api` application within Haystack that exposes Haystack's functionalities through a RESTful API.\n",
    "* **Docker** simplifies the environment setup needed to run Elasticsearch and Haystack API.\n",
    "\n",
    "> You can find ready-made base files (*document-search.haystack-pipeline.yml* and *docker-compose.yml*) for this tutorial in the [document-search-demo](https://github.com/bilgeyucel/document-search-demo) repository. After following the instructions in the repository, you can proceed directly to the ['Indexing Files to Elasticsearch'](#indexing-files-to-elasticsearch) section of this tutorial and begin indexing your documents."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Environment\n",
    "\n",
    "1. Install [Docker Compose](https://docs.docker.com/compose/) and launch Docker.\n",
    "If you installed Docker Desktop, just start the application. Run `docker info` to see if Docker is up and running:\n",
    "\n",
    "   ```bash\n",
    "   docker info\n",
    "   ```\n",
    "\n",
    "2. Download the *docker-compose.yml* file. Haystack provides a *docker-compose.yml* file that defines services for Haystack API and Elasticsearch. \n",
    "    1. Create a new folder called *doc-search* in a directory where you want to keep all tutorial related files.\n",
    "    2. Save the latest [*docker-compose.yml*](https://github.com/deepset-ai/haystack/blob/main/docker-compose.yml) file from GitHub into the folder. To save the *docker-compose.yml* file into the directory directly, run:\n",
    "\n",
    "         ```bash\n",
    "         curl --output docker-compose.yml https://raw.githubusercontent.com/deepset-ai/haystack/main/docker-compose.yml\n",
    "         ```\n",
    "\n",
    "    Here's what the */doc-search* folder should look like:\n",
    "    ```\n",
    "    /doc-search\n",
    "    └── docker-compose.yml\n",
    "    ```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that your environment's ready, you can start creating your indexing and query pipelines."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the Pipeline YAML File\n",
    "\n",
    "You can define components and pipelines using YAML code that Haystack translates into Python objects. In a pipeline YAML file, the `components` section lists all pipeline nodes and the `pipelines` section defines how these nodes are connected to each other. Let's start with defining two different pipelines, one to index your documents and another one to query them. We'll use one YAML file to define both pipelines.\n",
    "\n",
    "1. Create a document search pipeline. This will be your query pipeline:\n",
    "   1. In the newly created *doc-search* folder, create a file named *document-search.haystack-pipeline.yml*. The *docker-compose.yml* file and the new pipeline YAML file should be on the same level in the directory:\n",
    "\n",
    "      ```\n",
    "      /doc-search\n",
    "      ├── docker-compose.yml\n",
    "      └── document-search.haystack-pipeline.yml\n",
    "      ```\n",
    "\n",
    "   2. Provide the path to *document-search.haystack-pipeline.yml* as the `volume` source value in the *docker-compose.yml* file. The path must be relative to *docker-compose.yml*. As both files are in the same directory, the source value will be `./`. \n",
    "\n",
    "      ```yaml\n",
    "      haystack-api:\n",
    "        ...\n",
    "        volumes:\n",
    "          - ./:/opt/pipelines\n",
    "      ```\n",
    "\n",
    "   3. Update the `PIPELINE_YAML_PATH` variable in *docker-compose.yml* with the name of the pipeline YAML file. The `PIPELINE_YAML_PATH` variable tells `rest_api` which YAML file to run. \n",
    "\n",
    "      ```yaml\n",
    "      environment:\n",
    "        ...\n",
    "        - PIPELINE_YAML_PATH=/opt/pipelines/document-search.haystack-pipeline.yml\n",
    "        ...\n",
    "      ```\n",
    "   4. Define the pipeline nodes in the `components` section of the file. A document search pipeline requires a DocumentStore and a Retriever. Our pipeline will use `ElasticsearchDocumentStore` and `EmbeddingRetriever`:\n",
    "\n",
    "      ```yaml\n",
    "      components:\n",
    "        - name: DocumentStore # How you want to call this node here\n",
    "          type: ElasticsearchDocumentStore # This is the Haystack node class\n",
    "          params: # The node parameters\n",
    "            embedding_dim: 384 # This parameter is required for the embedding_model\n",
    "        - name: Retriever\n",
    "          type: EmbeddingRetriever\n",
    "          params:\n",
    "            document_store: DocumentStore\n",
    "            top_k: 10\n",
    "            embedding_model: sentence-transformers/all-MiniLM-L6-v2\n",
    "      ```\n",
    "\n",
    "   5. Create a query pipeline in the `pipelines` section. Here, `name` refers to the name of the pipeline, and `nodes` defines the order of the nodes in the pipeline: \n",
    "\n",
    "      ```yaml\n",
    "      pipelines:\n",
    "        - name: query \n",
    "          nodes:\n",
    "            - name: Retriever\n",
    "              inputs: [Query]\n",
    "      ```\n",
    "\n",
    "2. In the same YAML file, create an indexing pipeline. This pipeline will index your documents to Elasticsearch through `rest_api`. \n",
    "   1. Define `FileTypeClassifier`, `TextConverter`, and `PreProcessor` nodes for the pipeline:\n",
    "\n",
    "      ```yaml\n",
    "      components:\n",
    "        ...\n",
    "        - name: FileTypeClassifier\n",
    "          type: FileTypeClassifier\n",
    "        - name: TextFileConverter\n",
    "          type: TextConverter\n",
    "        - name: Preprocessor\n",
    "          type: PreProcessor\n",
    "          params: # These parameters define how you want to split your documents\n",
    "            split_by: word\n",
    "            split_length: 250\n",
    "            split_overlap: 30 \n",
    "            split_respect_sentence_boundary: True \n",
    "      ```\n",
    "\n",
    "   2. In the `pipelines` section of the YAML file, create a new pipeline called `indexing`. In this pipeline, indicate how the nodes you just defined are connected to each other, Retriever, and DocumentStore. This indexing pipeline supports *.TXT* files and preprocesses them before loading to Elasticsearch.\n",
    "\n",
    "      ```yaml\n",
    "      pipelines:\n",
    "        ...\n",
    "        - name: indexing\n",
    "          nodes:\n",
    "            - name: FileTypeClassifier\n",
    "              inputs: [File]\n",
    "            - name: TextFileConverter\n",
    "              inputs: [FileTypeClassifier.output_1]\n",
    "            - name: Preprocessor\n",
    "              inputs: [TextFileConverter]\n",
    "            - name: Retriever\n",
    "              inputs: [Preprocessor]\n",
    "            - name: DocumentStore\n",
    "              inputs: [Retriever]\n",
    "      ```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. After creating query and indexing pipelines, add `version: 1.19.0` to the top of the file. This is the Haystack version that comes with the Docker image in the *docker-compose.yml*. Now, the pipeline YAML is ready.\n",
    "\n",
    "```yaml\n",
    "version: 1.19.0\n",
    "\n",
    "components:\n",
    "  - name: DocumentStore\n",
    "    type: ElasticsearchDocumentStore\n",
    "    params:\n",
    "      embedding_dim: 384\n",
    "  - name: Retriever\n",
    "    type: EmbeddingRetriever\n",
    "    params:\n",
    "      document_store: DocumentStore\n",
    "      top_k: 10 \n",
    "      embedding_model: sentence-transformers/all-MiniLM-L6-v2\n",
    "  - name: FileTypeClassifier\n",
    "    type: FileTypeClassifier\n",
    "  - name: TextFileConverter\n",
    "    type: TextConverter\n",
    "  - name: Preprocessor\n",
    "    type: PreProcessor\n",
    "    params:\n",
    "      split_by: word\n",
    "      split_length: 250\n",
    "      split_overlap: 30 \n",
    "      split_respect_sentence_boundary: True\n",
    "\n",
    "pipelines:\n",
    "  - name: query \n",
    "    nodes:\n",
    "      - name: Retriever\n",
    "        inputs: [Query]\n",
    "  - name: indexing\n",
    "    nodes:\n",
    "      - name: FileTypeClassifier\n",
    "        inputs: [File]\n",
    "      - name: TextFileConverter\n",
    "        inputs: [FileTypeClassifier.output_1]\n",
    "      - name: Preprocessor\n",
    "        inputs: [TextFileConverter]\n",
    "      - name: Retriever\n",
    "        inputs: [Preprocessor]\n",
    "      - name: DocumentStore\n",
    "        inputs: [Retriever]\n",
    "```\n",
    "\n",
    "Feel free to play with the pipeline setup later on. Add or remove some nodes, change the parameters, or add new ones. For more options for nodes and parameters, check out [Haystack API Reference](https://docs.haystack.deepset.ai/reference/answer-generator-api)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launching Haystack API and Elasticsearch\n",
    "\n",
    "Pipelines are ready. Now it's time to start Elasticsearch and Haystack API.\n",
    "1. Run `docker-compose up` to start the `elasticsearch` and `haystack-api` containers. This command installs all the necessary packages, sets up the environment, and launches both Elasticsearch and Haystack API. Launching may take 2-3 minutes. \n",
    "\n",
    "   ```bash\n",
    "   docker-compose up\n",
    "   ```\n",
    "\n",
    "2. Test if everything is OK with the Haystack API by sending a cURL request to the `/initialized` endpoint. If everything works fine, you will get `true` as a response.\n",
    "\n",
    "   ```bash\n",
    "   curl --request GET http:https://127.0.0.1:8000/initialized\n",
    "   ```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Both containers are initialized. Time to fill your DocumentStore with files.  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing Files to Elasticsearch\n",
    "\n",
    "Right now, your Elasticsearch instance is empty. Haystack API provides a `/file-upload` endpoint to upload files to Elasticsearch. This endpoint uses the indexing pipeline you defined in the pipeline YAML. After indexing files to Elasticsearch, you can perform document search.\n",
    "\n",
    "1. Download the [example files](https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/article_txt_countries_and_capitals.zip) to the *doc-search* folder. The .zip file contains text files about countries and capitals crawled from [Wikipedia](https://en.wikipedia.org/wiki/Category:Lists_of_countries_by_continent).\n",
    "\n",
    "   ```\n",
    "   /doc-search\n",
    "   ├── docker-compose.yml\n",
    "   ├── document-search.haystack-pipeline.yml\n",
    "   └── /article_txt_countries_and_capitals\n",
    "       ├── 0_Minsk.txt\n",
    "       └── ...\n",
    "   ```\n",
    "\n",
    "2. Index files to Elasticsearch. You can send cURL requests to the `/file-upload` endpoint to upload files to the Elasticsearch instance. If the file is successfully uploaded, you will get `null` as a response.\n",
    "\n",
    "   ```bash\n",
    "   curl --request POST \\\n",
    "        --url http:https://127.0.0.1:8000/file-upload \\\n",
    "        --header 'accept: application/json' \\\n",
    "        --header 'content-type: multipart/form-data' \\\n",
    "        --form files=@article_txt_countries_and_capitals/0_Minsk.txt \\\n",
    "        --form meta=null\n",
    "   ```\n",
    "\n",
    "   This method is not the best one if you have multiple files to upload. That's because you need to replace file names in the request by hand. Instead, you can run a command that takes all *.TXT* files in the *article_txt_countries_and_capitals* folder and sends a POST request to index each file:   \n",
    "\n",
    "   ```bash\n",
    "   find ./article_txt_countries_and_capitals -name '*.txt' -exec \\\n",
    "        curl --request POST \\\n",
    "             --url http:https://127.0.0.1:8000/file-upload \\\n",
    "             --header 'accept: application/json' \\\n",
    "             --header 'content-type: multipart/form-data' \\\n",
    "             --form files=\"@{}\" \\\n",
    "             --form meta=null \\;\n",
    "   ```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying Your Pipeline\n",
    "\n",
    "That's it, the application is ready! Send another POST request to retrieve documents about _\"climate in Scandinavia\"_: \n",
    "\n",
    "```bash\n",
    "curl --request POST \\\n",
    "     --url http:https://127.0.0.1:8000/query \\\n",
    "     --header 'accept: application/json' \\\n",
    "     --header 'content-type: application/json' \\\n",
    "     --data '{\n",
    "     \"query\": \"climate in Scandinavia\"\n",
    "     }'\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a response, you will get a `QueryResponse` object consisting of `query`, `answers`, and `documents`. Documents related to your query will be under the `documents` attribute of the object.\n",
    "\n",
    "```python\n",
    "{\n",
    "  \"query\": \"climate in Scandinavia\",\n",
    "  \"answers\": [],\n",
    "  \"documents\": [\n",
    "    {\n",
    "      \"id\": \"24904f783ea4b90a47c33434a3e9df7a\",\n",
    "      \"content\": \"Because of Sweden's high latitude, the length of daylight varies greatly. North of the Arctic Circle, the sun never sets for part of each summer, and it never rises for part of each winter. In the capital, Stockholm, daylight lasts for more than 18 hours in late June but only around 6 hours in late December. Sweden receives between 1,100 and 1,900 hours of sunshine annually...\",\n",
    "      \"content_type\": \"text\",\n",
    "      \"meta\": {\n",
    "        \"_split_id\": 33,\n",
    "        \"name\": \"43_Sweden.txt\"\n",
    "      },\n",
    "      \"score\": 0.5017639926813274\n",
    "    },\n",
    "    ...\n",
    "  ]\n",
    "}\n",
    "```  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have created a proper search system that runs using Haystack REST API."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.6 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}