notebooks/vertexai-gemini-examples.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b2s1IpH6OGZQ"
      },
      "source": [
        "# Haystack 2.0 💙 Google Gemini\n",
        "\n",
        "*by Tuana Celik: [Twitter](https://twitter.com/tuanacelik), [LinkedIn](https://www.linkedin.com/in/tuanacelik/), Tilde Thurium: [Twitter](https://twitter.com/annthurium), [LinkedIn](https://www.linkedin.com/in/annthurium/) and Silvano Cerza: [LinkedIn](https://www.linkedin.com/in/silvanocerza/)*\n",
        "\n",
        "**📚 Check out the [Gemini Models with Google Vertex AI Integration for Haystack](https://haystack.deepset.ai/blog/gemini-models-with-google-vertex-for-haystack) article for a detailed run through of this example.**\n",
        "\n",
        "This is a notebook showing how you can use Gemini with Haystack 2.0.\n",
        "\n",
        "Gemini is Google's newest model. You can read more about its capabilities [here](https://deepmind.google/technologies/gemini/#capabilities).\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "toc",
        "id": "kWU_3yqtloif"
      },
      "source": [
        ">[Haystack 2.0 💙 Google Gemini](#scrollTo=b2s1IpH6OGZQ)\n",
        "\n",
        ">>[Install dependencies](#scrollTo=XromVwB1nQ76)\n",
        "\n",
        ">>[Use gemini-pro-vision](#scrollTo=08TA9zAQlqy6)\n",
        "\n",
        ">>>[Answer Questions](#scrollTo=c1ynAXT5mI1s)\n",
        "\n",
        ">>>[Answer Questions about Images](#scrollTo=H3pU-t_7mPyH)\n",
        "\n",
        ">>[Function Calling with gemini-pro](#scrollTo=gLIIQ4PZmX-H)\n",
        "\n",
        ">>[Build a full Retrieval-Augmented Generation Pipeline with gemini-pro-vision](#scrollTo=Pi37EVlDenPw)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XromVwB1nQ76"
      },
      "source": [
        "## Install dependencies\n",
        "\n",
        "As a prerequisite, you need to have a Google Cloud Project set up that has access to Gemini. Following that, you'll only need to authenticate yourself in this Colab.\n",
        "\n",
        "First thing first we need to install our dependencies.\n",
        "\n",
        "(You can ignore the pip dependency error for `cohere` and `tiktoken`, that's irrelevant for our purposes.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wxGffegfOGZR"
      },
      "outputs": [],
      "source": [
        "!pip install --upgrade haystack-ai google-vertex-haystack"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tHDiEzI2OGZU"
      },
      "source": [
        "To use Gemini you need to have a Google Cloud Platform account and be logged in using Application Default Credentials (ADCs). For more info see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).\n",
        "\n",
        "Time to login!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iKvKRuRXOGZU"
      },
      "outputs": [],
      "source": [
        "from google.colab import auth\n",
        "\n",
        "auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5uw5F0M_OGZT"
      },
      "source": [
        "Remember to set the `project_id` variable to a valid project ID that you have enough authorization to use for Gemini.\n",
        "We're going to use this one throughout the example!\n",
        "\n",
        "To find your project ID you can find it in the [GCP resource manager](https://console.cloud.google.com/cloud-resource-manager) or locally by running `gcloud projects list` in your terminal. For more info on the gcloud CLI see the [official documentation](https://cloud.google.com/cli)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VzA_x7iFOGZT"
      },
      "outputs": [],
      "source": [
        "project_id = input(\"Enter your project ID:\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08TA9zAQlqy6"
      },
      "source": [
        "## Use `gemini-pro-vision`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c1ynAXT5mI1s"
      },
      "source": [
        "### Answer Questions\n",
        "\n",
        "Now that we setup everything we can create an instance of our Gemini component."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f6Ql3qSlOGZV"
      },
      "outputs": [],
      "source": [
        "from haystack_integrations.components.generators.google_vertex import VertexAIGeminiGenerator\n",
        "\n",
        "gemini = VertexAIGeminiGenerator(model=\"gemini-pro-vision\", project_id=project_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0qRBY_hGOGZV"
      },
      "source": [
        "Let's start by asking something simple.\n",
        "\n",
        "This component expects a list of `Parts` as input to the `run()` method. Parts can be anything from a message, to images, or even function calls. Here are the docstrings from the source code for the most up-to-date reference we could find [here.](https://github.com/googleapis/python-aiplatform/blob/5f6ad8df5a08e78a121a72a21e21d95abb072e58/vertexai/generative_models/_generative_models.py#L1427-L1446)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QbqFt4IiOGZV"
      },
      "outputs": [],
      "source": [
        "result = gemini.run(parts = [\"What is the most interesting thing you know?\"])\n",
        "for answer in result[\"answers\"]:\n",
        "    print(answer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H3pU-t_7mPyH"
      },
      "source": [
        "### Answer Questions about Images"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-wi810nOGZW"
      },
      "source": [
        "Let's try something a bit different! `gemini-pro-vision` can also work with images, let's see if we can have it answer questions about some robots 👇\n",
        "\n",
        "We're going to download some images for this example. 🤖"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sGqQ8hkhOGZW"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "from haystack.dataclasses.byte_stream import ByteStream\n",
        "\n",
        "URLS = [\n",
        "    \"https://raw.githubusercontent.com/silvanocerza/robots/main/robot1.jpg\",\n",
        "    \"https://raw.githubusercontent.com/silvanocerza/robots/main/robot2.jpg\",\n",
        "    \"https://raw.githubusercontent.com/silvanocerza/robots/main/robot3.jpg\",\n",
        "    \"https://raw.githubusercontent.com/silvanocerza/robots/main/robot4.jpg\"\n",
        "]\n",
        "images = [\n",
        "    ByteStream(data=requests.get(url).content, mime_type=\"image/jpeg\")\n",
        "    for url in URLS\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tQsdovv2pWzh"
      },
      "source": [
        "Next, let's run the `VertexAIGeminiGenerator` component on it's own."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6jSBGEkmpPsN"
      },
      "outputs": [],
      "source": [
        "result = gemini.run(parts = [\"What can you tell me about this robots?\", *images])\n",
        "for answer in result[\"answers\"]:\n",
        "    print(answer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Qp9tGaNOGZW"
      },
      "source": [
        "Did Gemini recognize all its friends? 👀"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gLIIQ4PZmX-H"
      },
      "source": [
        "## Function Calling with `gemini-pro`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iCiSz840mfME"
      },
      "source": [
        "With `gemini-pro`, we can also start introducing function calling!\n",
        "So let's see how we can do that 👇\n",
        "\n",
        "Let's see if we can build a system that can run a `get_current_weather` function, based on a question asked in natural language.\n",
        "\n",
        "First we create our function definition and tool.\n",
        "\n",
        "For demonstration purposes, we're simply creating a `get_current_weather` function that returns an object which will _always_ tell us it's 'Sunny, and 21.8 degrees'.. If it's Celsius, that's a good day! ☀️"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4Wa_IoDDNg9V"
      },
      "outputs": [],
      "source": [
        "def get_current_weather(location: str, unit: str = \"celsius\"):\n",
        "  return {\"weather\": \"sunny\", \"temperature\": 21.8, \"unit\": unit}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GjapaK2yq-ce"
      },
      "source": [
        "Now we have to provide this function as a `Tool` to Gemini. So, first we need to create a `FunctionDeclaration` that _explains_ this function to Gemini 👇"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6KCv7g4JOGZW"
      },
      "outputs": [],
      "source": [
        "from vertexai.preview.generative_models import Tool, FunctionDeclaration\n",
        "\n",
        "get_current_weather_func = FunctionDeclaration(\n",
        "    name=\"get_current_weather\",\n",
        "    description=\"Get the current weather in a given location\",\n",
        "    parameters={\n",
        "        \"type\": \"object\",\n",
        "        \"properties\": {\n",
        "            \"location\": {\"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\"},\n",
        "            \"unit\": {\n",
        "                \"type\": \"string\",\n",
        "                \"enum\": [\n",
        "                    \"celsius\",\n",
        "                    \"fahrenheit\",\n",
        "                ],\n",
        "            },\n",
        "        },\n",
        "        \"required\": [\"location\"],\n",
        "    },\n",
        ")\n",
        "tool = Tool([get_current_weather_func])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TlbWFohMOGZW"
      },
      "source": [
        "We're also going to chat with Gemini this time, we're going to use another class for this.\n",
        "\n",
        "We also need the Gemini Pro model to use functions, Gemini Pro Vision doesn't support functions.\n",
        "\n",
        "Let's create a `VertexAIGeminiChatGenerator`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uPIPj6HsOGZW"
      },
      "outputs": [],
      "source": [
        "from haystack_integrations.components.generators.google_vertex import VertexAIGeminiChatGenerator\n",
        "\n",
        "gemini_chat = VertexAIGeminiChatGenerator(model=\"gemini-pro\", project_id=project_id, tools=[tool])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HD4G61Z0OGZX"
      },
      "outputs": [],
      "source": [
        "from haystack.dataclasses import ChatMessage\n",
        "\n",
        "messages = [ChatMessage.from_user(content = \"What is the temperature in celsius in Berlin?\")]\n",
        "res = gemini_chat.run(messages=messages)\n",
        "res[\"replies\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bbEpArmfOGZX"
      },
      "source": [
        "Look at that! We go a message with some interesting information now.\n",
        "We can use that information to call a real function locally.\n",
        "\n",
        "Let's do exactly that and pass the result back to Gemini."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c-DurWKOOSOk"
      },
      "outputs": [],
      "source": [
        "\n",
        "weather = get_current_weather(**res[\"replies\"][0].content)\n",
        "\n",
        "messages += res[\"replies\"] + [ChatMessage.from_function(content=weather, name=\"get_current_weather\")]\n",
        "\n",
        "res = gemini_chat.run(messages = messages)\n",
        "res[\"replies\"][0].content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YsiwN7dbOwSW"
      },
      "source": [
        "Seems like the weather is nice and sunny, remember to put on your sunglasses. 😎"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pi37EVlDenPw"
      },
      "source": [
        "## Build a full Retrieval-Augmented Generation Pipeline with `gemini-pro-vision`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQz9_N46hniU"
      },
      "source": [
        "As a final exercise, let's add the `VertexAIGeminiGenerator` to a full RAG pipeline. In the example below, we are building a RAG pipeline that does question answering on the web, using `gemini-pro-vision`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "43QF7y1PhRQn"
      },
      "outputs": [],
      "source": [
        "from haystack.components.fetchers.link_content import LinkContentFetcher\n",
        "from haystack.components.converters import HTMLToDocument\n",
        "from haystack.components.preprocessors import DocumentSplitter\n",
        "from haystack.components.rankers import TransformersSimilarityRanker\n",
        "from haystack.components.builders.prompt_builder import PromptBuilder\n",
        "from haystack import Pipeline\n",
        "\n",
        "fetcher = LinkContentFetcher()\n",
        "converter = HTMLToDocument()\n",
        "document_splitter = DocumentSplitter(split_by=\"word\", split_length=50)\n",
        "similarity_ranker = TransformersSimilarityRanker(top_k=3)\n",
        "gemini = VertexAIGeminiGenerator(model=\"gemini-pro-vision\", project_id=project_id)\n",
        "\n",
        "prompt_template = \"\"\"\n",
        "According to these documents:\n",
        "\n",
        "{% for doc in documents %}\n",
        "  {{ doc.content }}\n",
        "{% endfor %}\n",
        "\n",
        "Answer the given question: {{question}}\n",
        "Answer:\n",
        "\"\"\"\n",
        "prompt_builder = PromptBuilder(template=prompt_template)\n",
        "\n",
        "pipeline = Pipeline()\n",
        "pipeline.add_component(\"fetcher\", fetcher)\n",
        "pipeline.add_component(\"converter\", converter)\n",
        "pipeline.add_component(\"splitter\", document_splitter)\n",
        "pipeline.add_component(\"ranker\", similarity_ranker)\n",
        "pipeline.add_component(\"prompt_builder\", prompt_builder)\n",
        "pipeline.add_component(\"gemini\", gemini)\n",
        "\n",
        "pipeline.connect(\"fetcher.streams\", \"converter.sources\")\n",
        "pipeline.connect(\"converter.documents\", \"splitter.documents\")\n",
        "pipeline.connect(\"splitter.documents\", \"ranker.documents\")\n",
        "pipeline.connect(\"ranker.documents\", \"prompt_builder.documents\")\n",
        "pipeline.connect(\"prompt_builder.prompt\", \"gemini\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cGqt94B0fZi1"
      },
      "source": [
        "Let's try asking Gemini to tell us about Haystack 2.0 and how to use it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EhEx8xO7jMf9"
      },
      "outputs": [],
      "source": [
        "question = \"What do graphs have to do with Haystack?\"\n",
        "result = pipeline.run({\"prompt_builder\": {\"question\": question},\n",
        "                   \"ranker\": {\"query\": question},\n",
        "                   \"fetcher\": {\"urls\": [\"https://haystack.deepset.ai/blog/introducing-haystack-2-beta-and-advent\"]}})\n",
        "\n",
        "for answer in result[\"gemini\"][\"answers\"]:\n",
        "  print(answer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xk2SAGT_l25K"
      },
      "source": [
        "Now you've seen some of what Gemini can do, as well as how to integrate it with Haystack 2.0. If you want to learn more:\n",
        "- check out the Haystack [docs](https://docs.haystack.deepset.ai/docs) or [tutorials](https://haystack.deepset.ai/tutorials)\n",
        "- Try out the [Gemini quickstart colab from Google](https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/tutorials/python_quickstart.ipynb#scrollTo=IqFXdgDFRvlU)\n",
        "- Participate in the [Advent of Haystack](https://haystack.deepset.ai/advent-of-haystack)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "google-vertex-haystack",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}