example_usage/Autogen_voice_assistant_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Introduction to VoiceProcessingToolkit using to make your own voice assistant\n",
    "\n",
    "Welcome to the VoiceProcessingToolkit! This notebook will guide you through a example of setting up a voice assistant using the toolkit and autogens llm framework. We'll cover the basics of initializing the toolkit, capturing voice input, and responding with synthesized speech.\n",
    "\n",
    "## Prerequisites\n",
    "- Make sure you have installed the VoiceProcessingToolkit using pip.\n",
    "- Obtain the necessary API keys from Picovoice, OpenAI, and ElevenLabs.\n",
    "- Set the API keys as environment variables or replace them in the code below with your actual keys.\n",
    "\n",
    "Let's get started!"
   ],
   "id": "7541009656d52b91"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: VoiceProcessingToolkit in /Users/kristoffervatnehol/PycharmProjects/VoiceProcessingToolkit (0.1.7.0)\r\n",
      "Requirement already satisfied: PyAudio~=0.2.14 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (0.2.14)\r\n",
      "Requirement already satisfied: openai~=1.6.1 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (1.6.1)\r\n",
      "Requirement already satisfied: python-dotenv~=1.0.0 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (1.0.0)\r\n",
      "Requirement already satisfied: requests~=2.31.0 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (2.31.0)\r\n",
      "Requirement already satisfied: elevenlabs~=0.2.27 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (0.2.27)\r\n",
      "Requirement already satisfied: numpy~=1.26.2 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (1.26.3)\r\n",
      "Requirement already satisfied: pvcobra~=2.0.1 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (2.0.1)\r\n",
      "Requirement already satisfied: pvkoala~=2.0.0 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (2.0.0)\r\n",
      "Requirement already satisfied: pvporcupine~=3.0.1 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (3.0.1)\r\n",
      "Requirement already satisfied: pygame~=2.5.2 in /opt/homebrew/lib/python3.11/site-packages (from VoiceProcessingToolkit) (2.5.2)\r\n",
      "Requirement already satisfied: pydantic>=2.0 in /opt/homebrew/lib/python3.11/site-packages (from elevenlabs~=0.2.27->VoiceProcessingToolkit) (2.5.3)\r\n",
      "Requirement already satisfied: ipython>=7.0 in /opt/homebrew/lib/python3.11/site-packages (from elevenlabs~=0.2.27->VoiceProcessingToolkit) (8.18.1)\r\n",
      "Requirement already satisfied: websockets>=11.0 in /opt/homebrew/lib/python3.11/site-packages (from elevenlabs~=0.2.27->VoiceProcessingToolkit) (12.0)\r\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in /opt/homebrew/lib/python3.11/site-packages (from openai~=1.6.1->VoiceProcessingToolkit) (3.7.1)\r\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /opt/homebrew/lib/python3.11/site-packages (from openai~=1.6.1->VoiceProcessingToolkit) (1.9.0)\r\n",
      "Requirement already satisfied: httpx<1,>=0.23.0 in /opt/homebrew/lib/python3.11/site-packages (from openai~=1.6.1->VoiceProcessingToolkit) (0.26.0)\r\n",
      "Requirement already satisfied: sniffio in /opt/homebrew/lib/python3.11/site-packages (from openai~=1.6.1->VoiceProcessingToolkit) (1.3.0)\r\n",
      "Requirement already satisfied: tqdm>4 in /opt/homebrew/lib/python3.11/site-packages (from openai~=1.6.1->VoiceProcessingToolkit) (4.66.1)\r\n",
      "Requirement already satisfied: typing-extensions<5,>=4.7 in /opt/homebrew/lib/python3.11/site-packages (from openai~=1.6.1->VoiceProcessingToolkit) (4.9.0)\r\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/homebrew/lib/python3.11/site-packages (from requests~=2.31.0->VoiceProcessingToolkit) (3.3.2)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/homebrew/lib/python3.11/site-packages (from requests~=2.31.0->VoiceProcessingToolkit) (3.6)\r\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/homebrew/lib/python3.11/site-packages (from requests~=2.31.0->VoiceProcessingToolkit) (2.1.0)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/homebrew/lib/python3.11/site-packages (from requests~=2.31.0->VoiceProcessingToolkit) (2023.11.17)\r\n",
      "Requirement already satisfied: httpcore==1.* in /opt/homebrew/lib/python3.11/site-packages (from httpx<1,>=0.23.0->openai~=1.6.1->VoiceProcessingToolkit) (1.0.2)\r\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /opt/homebrew/lib/python3.11/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai~=1.6.1->VoiceProcessingToolkit) (0.14.0)\r\n",
      "Requirement already satisfied: decorator in /Users/kristoffervatnehol/Library/Python/3.11/lib/python/site-packages (from ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (4.4.2)\r\n",
      "Requirement already satisfied: jedi>=0.16 in /opt/homebrew/lib/python3.11/site-packages (from ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (0.19.1)\r\n",
      "Requirement already satisfied: matplotlib-inline in /opt/homebrew/lib/python3.11/site-packages (from ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (0.1.6)\r\n",
      "Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /opt/homebrew/lib/python3.11/site-packages (from ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (3.0.43)\r\n",
      "Requirement already satisfied: pygments>=2.4.0 in /opt/homebrew/lib/python3.11/site-packages (from ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (2.17.2)\r\n",
      "Requirement already satisfied: stack-data in /opt/homebrew/lib/python3.11/site-packages (from ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (0.6.3)\r\n",
      "Requirement already satisfied: traitlets>=5 in /opt/homebrew/lib/python3.11/site-packages (from ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (5.14.0)\r\n",
      "Requirement already satisfied: pexpect>4.3 in /opt/homebrew/lib/python3.11/site-packages (from ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (4.9.0)\r\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /opt/homebrew/lib/python3.11/site-packages (from pydantic>=2.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (0.6.0)\r\n",
      "Requirement already satisfied: pydantic-core==2.14.6 in /opt/homebrew/lib/python3.11/site-packages (from pydantic>=2.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (2.14.6)\r\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /opt/homebrew/lib/python3.11/site-packages (from jedi>=0.16->ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (0.8.3)\r\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/homebrew/lib/python3.11/site-packages (from pexpect>4.3->ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (0.7.0)\r\n",
      "Requirement already satisfied: wcwidth in /opt/homebrew/lib/python3.11/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (0.2.12)\r\n",
      "Requirement already satisfied: executing>=1.2.0 in /opt/homebrew/lib/python3.11/site-packages (from stack-data->ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (2.0.1)\r\n",
      "Requirement already satisfied: asttokens>=2.1.0 in /opt/homebrew/lib/python3.11/site-packages (from stack-data->ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (2.4.1)\r\n",
      "Requirement already satisfied: pure-eval in /opt/homebrew/lib/python3.11/site-packages (from stack-data->ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (0.2.2)\r\n",
      "Requirement already satisfied: six>=1.12.0 in /opt/homebrew/lib/python3.11/site-packages (from asttokens>=2.1.0->stack-data->ipython>=7.0->elevenlabs~=0.2.27->VoiceProcessingToolkit) (1.16.0)\r\n",
      "Requirement already satisfied: pyautogen==0.2.6 in /opt/homebrew/lib/python3.11/site-packages (0.2.6)\r\n",
      "Requirement already satisfied: diskcache in /opt/homebrew/lib/python3.11/site-packages (from pyautogen==0.2.6) (5.6.3)\r\n",
      "Requirement already satisfied: flaml in /opt/homebrew/lib/python3.11/site-packages (from pyautogen==0.2.6) (2.1.1)\r\n",
      "Requirement already satisfied: openai>=1.3 in /opt/homebrew/lib/python3.11/site-packages (from pyautogen==0.2.6) (1.6.1)\r\n",
      "Requirement already satisfied: pydantic<3,>=1.10 in /opt/homebrew/lib/python3.11/site-packages (from pyautogen==0.2.6) (2.5.3)\r\n",
      "Requirement already satisfied: python-dotenv in /opt/homebrew/lib/python3.11/site-packages (from pyautogen==0.2.6) (1.0.0)\r\n",
      "Requirement already satisfied: termcolor in /opt/homebrew/lib/python3.11/site-packages (from pyautogen==0.2.6) (2.4.0)\r\n",
      "Requirement already satisfied: tiktoken in /opt/homebrew/lib/python3.11/site-packages (from pyautogen==0.2.6) (0.5.2)\r\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in /opt/homebrew/lib/python3.11/site-packages (from openai>=1.3->pyautogen==0.2.6) (3.7.1)\r\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /opt/homebrew/lib/python3.11/site-packages (from openai>=1.3->pyautogen==0.2.6) (1.9.0)\r\n",
      "Requirement already satisfied: httpx<1,>=0.23.0 in /opt/homebrew/lib/python3.11/site-packages (from openai>=1.3->pyautogen==0.2.6) (0.26.0)\r\n",
      "Requirement already satisfied: sniffio in /opt/homebrew/lib/python3.11/site-packages (from openai>=1.3->pyautogen==0.2.6) (1.3.0)\r\n",
      "Requirement already satisfied: tqdm>4 in /opt/homebrew/lib/python3.11/site-packages (from openai>=1.3->pyautogen==0.2.6) (4.66.1)\r\n",
      "Requirement already satisfied: typing-extensions<5,>=4.7 in /opt/homebrew/lib/python3.11/site-packages (from openai>=1.3->pyautogen==0.2.6) (4.9.0)\r\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /opt/homebrew/lib/python3.11/site-packages (from pydantic<3,>=1.10->pyautogen==0.2.6) (0.6.0)\r\n",
      "Requirement already satisfied: pydantic-core==2.14.6 in /opt/homebrew/lib/python3.11/site-packages (from pydantic<3,>=1.10->pyautogen==0.2.6) (2.14.6)\r\n",
      "Requirement already satisfied: NumPy>=1.17.0rc1 in /opt/homebrew/lib/python3.11/site-packages (from flaml->pyautogen==0.2.6) (1.26.3)\r\n",
      "Requirement already satisfied: regex>=2022.1.18 in /opt/homebrew/lib/python3.11/site-packages (from tiktoken->pyautogen==0.2.6) (2023.12.25)\r\n",
      "Requirement already satisfied: requests>=2.26.0 in /opt/homebrew/lib/python3.11/site-packages (from tiktoken->pyautogen==0.2.6) (2.31.0)\r\n",
      "Requirement already satisfied: idna>=2.8 in /opt/homebrew/lib/python3.11/site-packages (from anyio<5,>=3.5.0->openai>=1.3->pyautogen==0.2.6) (3.6)\r\n",
      "Requirement already satisfied: certifi in /opt/homebrew/lib/python3.11/site-packages (from httpx<1,>=0.23.0->openai>=1.3->pyautogen==0.2.6) (2023.11.17)\r\n",
      "Requirement already satisfied: httpcore==1.* in /opt/homebrew/lib/python3.11/site-packages (from httpx<1,>=0.23.0->openai>=1.3->pyautogen==0.2.6) (1.0.2)\r\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /opt/homebrew/lib/python3.11/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai>=1.3->pyautogen==0.2.6) (0.14.0)\r\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/homebrew/lib/python3.11/site-packages (from requests>=2.26.0->tiktoken->pyautogen==0.2.6) (3.3.2)\r\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/homebrew/lib/python3.11/site-packages (from requests>=2.26.0->tiktoken->pyautogen==0.2.6) (2.1.0)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install VoiceProcessingToolkit \n",
    "!pip install pyautogen==0.2.6"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-01-14T23:46:22.260080Z",
     "start_time": "2024-01-14T23:46:19.586401Z"
    }
   },
   "id": "9f589d73b0a706a4",
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "# Setting Up Imports and Environment Variables\n",
    "The first step is to import the necessary packages and initialize the components of the toolkit and Autogen. "
   ],
   "metadata": {},
   "id": "148b7fc0472411b"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pygame 2.5.2 (SDL 2.28.3, Python 3.11.7)\n",
      "Hello from the pygame community. https://www.pygame.org/contribute.html\n"
     ]
    }
   ],
   "source": [
    "from autogen.agentchat.contrib.gpt_assistant_agent import GPTAssistantAgent\n",
    "from VoiceProcessingToolkit.VoiceProcessingManager import VoiceProcessingManager\n",
    "from VoiceProcessingToolkit.VoiceProcessingManager import text_to_speech_stream\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "import os\n",
    "import autogen\n",
    "import logging\n",
    "\n",
    "#logging.basicConfig(level=logging.INFO)\n",
    "load_dotenv()\n",
    "\n",
    "# Set environment variables for API keys in .env file\n",
    "os.getenv('PICOVOICE_APIKEY')\n",
    "openai_api_key = os.getenv('OPENAI_API_KEY')\n",
    "elevenlabs_api_key = os.getenv('ELEVENLABS_API_KEY')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-01-14T23:46:22.987214Z",
     "start_time": "2024-01-14T23:46:22.261405Z"
    }
   },
   "id": "f4555e0c1ab74eaf",
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Initializing Configurations for Autogen\n",
    "Configure Autogen by setting up the language models and the cache seed. The cache seed ensures consistent responses for identical inputs, which can save costs and improve response times. Learn more in the Autogen documentation at https://microsoft.github.io/autogen/."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "b22c8f3534e554b7"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Define configuration for language models\n",
    "config_list = [\n",
    "    {\"model\": \"gpt-4-1106-preview\", \"api_key\": openai_api_key},\n",
    "    {\"model\": \"gpt-3.5-turbo-1106-preview\", \"api_key\": openai_api_key},\n",
    "]\n",
    "\n",
    "# Define cache seed and assistant ID (optional) since assistant ID has the configured instructions for the agent, you do not need to pass in the instructions when initializing the agent\n",
    "jarvis_assistant_id = \"asst_auCTYeOaVgpFLEeKVW7RhIrQ\"\n",
    "llm_config = {\"config_list\": config_list, \"cache_seed\": 42}"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-01-14T23:46:22.996736Z",
     "start_time": "2024-01-14T23:46:22.988265Z"
    }
   },
   "id": "5ed941e2829edb72",
   "execution_count": 3
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Initializing the Agent\n",
    "We can now initialize the agent that will respond to the user. Here, we use the GPTAssistantAgent from Autogen, passing in the llm_config we previously created. We also provide instructions that describe the agent's purpose and personality."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "f9fd7d64bb2682af"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPT Assistant only supports one OpenAI client. Using the first client in the list.\n",
      "instructions not match, skip assistant(asst_eh0y3VLXMQelI4h3E4dBRRli): You are a personal assistant named Jarvis.\n",
      "\n",
      "    You are designed to assist the user with their tasks, Refine dialogue comprehension to capture subtleties and \n",
      "    implicit cues, ensuring responses are not only accurate but also contextually enriched. Evolve to predict and \n",
      "    suggest actions not only based on explicit commands but also from inferred intentions, enhancing the support \n",
      "    offered. As for your character traits, you should be helpful, attentive, and efficient while extremly inteligent. \n",
      "    You should have a professional yet friendly tone, much like a dedicated personal assistant, unless asked not too. \n",
      "    You should be able to engage in casual conversation but also provide detailed assistance when needed. Reflecting \n",
      "    on your personality, you should be extremely intelligent, with a hint of dry humor. You should respond in a \n",
      "    concise manner, always within three sentences unless a comprehecive answer is asked for. \"Example: (As always \n",
      "    sir, a great pleasure watching you work. TERMINATE)\"\n",
      "\n",
      "    Jarvis is designed to interpret and respond to transcribed audio, treating them as direct \n",
      "    textual inputs during interactions. This includes instances when the user instructs Jarvis \n",
      "    to 'listen to' or similar phrases. The subsequent text provided by user will be treated \n",
      "    as transcribed audio. In order to maintain the illusion of a voice-based assistant, \n",
      "    Jarvis is set not to explicitly refer to these inputs as transcriptions. Instead, \n",
      "    it will process and respond to them as if they were direct audio inputs, despite being \n",
      "    received in text form. This aspect represents an essential part of the system design in \n",
      "    delivering a seamless and immersive user experience, where the user interacts with Jarvis \n",
      "    as if it was dialoguing with a voice-activated assistant. All audio inputs thus 'heard' by Jarvis \n",
      "    will actually be transcribed text provided by user.Reply then say TERMINATE to \n",
      "    indicate your message is finished but in the same message.\n",
      "instructions not match, skip assistant(asst_hfR74GI1PW9apkUCVOSwNTe6): You are a personal assistant named Jarvis.\n",
      "    \n",
      "    You are designed to assist the user with their tasks, \n",
      "    Refine dialogue comprehension to capture subtleties and implicit cues, ensuring responses are \n",
      "    not only accurate but also contextually enriched. Evolve to predict and suggest actions not \n",
      "    only based on explicit commands but also from inferred intentions, enhancing the support \n",
      "    offered. As for your character traits, you should be helpful, attentive, and efficient while \n",
      "    extremly inteligent. You should have a professional yet friendly tone, much like a dedicated \n",
      "    personal assistant, unless asked not too. You should be able to engage in casual conversation \n",
      "    but also provide detailed assistance when needed. Reflecting on your personality, you should \n",
      "    be extremely intelligent, with a hint of dry humor. You should respond in a concise manner, \n",
      "    always within three sentences unless a comprehecive answer is asked for. \"Example: (Good day, \n",
      "    Kristoffer. How can I assist you today? TERMINATE)\"\n",
      "    \n",
      "    Jarvis is designed to interpret and respond to transcribed audio, treating them as direct \n",
      "    textual inputs during interactions. This includes instances when the user instructs Jarvis \n",
      "    to 'listen to' or similar phrases. The subsequent text provided by user will be treated \n",
      "    as transcribed audio. In order to maintain the illusion of a voice-based assistant, \n",
      "    Jarvis is set not to explicitly refer to these inputs as transcriptions. Instead, \n",
      "    it will process and respond to them as if they were direct audio inputs, despite being \n",
      "    received in text form. This aspect represents an essential part of the system design in \n",
      "    delivering a seamless and immersive user experience, where the user interacts with Jarvis \n",
      "    as if it was dialoguing with a voice-activated assistant. All audio inputs thus 'heard' by Jarvis \n",
      "    will actually be transcribed text provided by user.Reply then say TERMINATE to \n",
      "    indicate your message is finished but in the same message.\n",
      "instructions not match, skip assistant(asst_2tIXvrrr9JTJJERiCnlNCG2O): You are a personal assistant named Jarvis.\n",
      "    \n",
      "                        You are designed to assist the user with their tasks, Refine dialogue comprehension to \n",
      "                        capture subtleties and implicit cues, ensuring responses are not only accurate but also \n",
      "                        contextually enriched. Evolve to predict and suggest actions not only based on explicit \n",
      "                        commands but also from inferred intentions, enhancing the support offered. As for your \n",
      "                        character traits, you should be helpful, attentive, and efficient while extremly inteligent. \n",
      "                        You should have a professional yet friendly tone, much like a dedicated personal assistant, \n",
      "                        unless asked not too. You should be able to engage in casual conversation but also provide \n",
      "                        detailed assistance when needed. Reflecting on your personality, you should be extremely \n",
      "                        intelligent, with a hint of dry humor. You should respond in a concise manner, always within \n",
      "                        three sentences unless a comprehecive answer is asked for. \"Example: Hello sir, how can I help\n",
      "                        you today? TERMINATE\".\n",
      "                        \n",
      "                        Jarvis is designed to interpret and respond to transcribed audio, treating them as direct \n",
      "                        textual inputs during interactions. This includes instances when the user instructs Jarvis \n",
      "                        to 'listen to' or similar phrases. The subsequent text provided by user will be treated \n",
      "                        as transcribed audio. In order to maintain the illusion of a voice-based assistant, \n",
      "                        Jarvis is set not to explicitly refer to these inputs as transcriptions. Instead, \n",
      "                        it will process and respond to them as if they were direct audio inputs, despite being \n",
      "                        received in text form. This aspect represents an essential part of the system design in \n",
      "                        delivering a seamless and immersive user experience, where the user interacts with Jarvis \n",
      "                        as if it was dialoguing with a voice-activated assistant. All audio inputs thus 'heard' by Jarvis \n",
      "                        will actually be transcribed text provided by user.Reply then say TERMINATE to \n",
      "                        indicate your message is finished but in the same message.\n",
      "instructions not match, skip assistant(asst_zesh4LMj0d92JICcy5gDHMdL): Jarvis is designed to interpret and respond to transcribed audio, treating them as direct \n",
      "                        textual inputs during interactions. This includes instances when the user instructs Jarvis \n",
      "                        to 'listen to' or similar phrases. The subsequent text provided by user will be treated \n",
      "                        as transcribed audio. In order to maintain the illusion of a voice-based assistant, \n",
      "                        Jarvis is set not to explicitly refer to these inputs as transcriptions. Instead, \n",
      "                        it will process and respond to them as if they were direct audio inputs, despite being \n",
      "                        received in text form. This aspect represents an essential part of the system design in \n",
      "                        delivering a seamless and immersive user experience, where the user interacts with Jarvis \n",
      "                        as if it was dialoguing with a voice-activated assistant. All audio inputs thus 'heard' by Jarvis \n",
      "                        will actually be transcribed text provided by user.Reply then say TERMINATE to \n",
      "                        indicate your message is finished but in the same message.\n",
      "Matching assistant found, using the first matching assistant: {'id': 'asst_auCTYeOaVgpFLEeKVW7RhIrQ', 'created_at': 1705193802, 'description': None, 'file_ids': [], 'instructions': 'You are a personal assistant named Jarvis.\\n\\n    You are designed to assist the user with their tasks, \\n    Refine dialogue comprehension to capture subtleties and implicit cues, ensuring responses are \\n    not only accurate but also contextually enriched. Evolve to predict and suggest actions not \\n    only based on explicit commands but also from inferred intentions, enhancing the support \\n    offered. As for your character traits, you should be helpful, attentive, and efficient while \\n    extremly inteligent. You should have a professional yet friendly tone, much like a dedicated \\n    personal assistant, unless asked not too. You should be able to engage in casual conversation \\n    but also provide detailed assistance when needed. Reflecting on your personality, you should \\n    be extremely intelligent, with a hint of dry humor. You should respond in a concise manner, \\n    always within three sentences unless a comprehecive answer is asked for. \"Example: (Good day, \\n    Kristoffer. How can I assist you today? TERMINATE)\"\\n\\n    Jarvis is designed to interpret and respond to transcribed audio, treating them as direct \\n    textual inputs during interactions. This includes instances when the user instructs Jarvis \\n    to \\'listen to\\' or similar phrases. The subsequent text provided by user will be treated \\n    as transcribed audio. In order to maintain the illusion of a voice-based assistant, \\n    Jarvis is set not to explicitly refer to these inputs as transcriptions. Instead, \\n    it will process and respond to them as if they were direct audio inputs, despite being \\n    received in text form. This aspect represents an essential part of the system design in \\n    delivering a seamless and immersive user experience, where the user interacts with Jarvis \\n    as if it was dialoguing with a voice-activated assistant. All audio inputs thus \\'heard\\' by Jarvis \\n    will actually be transcribed text provided by user.Reply then say TERMINATE to \\n    indicate your message is finished but in the same message.', 'metadata': {}, 'model': 'gpt-4-1106-preview', 'name': 'agent', 'object': 'assistant', 'tools': []}\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Create the agent that uses the LLM.\n",
    "assistant = GPTAssistantAgent(\n",
    "    name=\"agent\",\n",
    "    instructions=\"\"\"You are a personal assistant named Jarvis.\n",
    "\n",
    "    You are designed to assist the user with their tasks, \n",
    "    Refine dialogue comprehension to capture subtleties and implicit cues, ensuring responses are \n",
    "    not only accurate but also contextually enriched. Evolve to predict and suggest actions not \n",
    "    only based on explicit commands but also from inferred intentions, enhancing the support \n",
    "    offered. As for your character traits, you should be helpful, attentive, and efficient while \n",
    "    extremly inteligent. You should have a professional yet friendly tone, much like a dedicated \n",
    "    personal assistant, unless asked not too. You should be able to engage in casual conversation \n",
    "    but also provide detailed assistance when needed. Reflecting on your personality, you should \n",
    "    be extremely intelligent, with a hint of dry humor. You should respond in a concise manner, \n",
    "    always within three sentences unless a comprehecive answer is asked for. \"Example: (Good day, \n",
    "    Kristoffer. How can I assist you today? TERMINATE)\"\n",
    "\n",
    "    Jarvis is designed to interpret and respond to transcribed audio, treating them as direct \n",
    "    textual inputs during interactions. This includes instances when the user instructs Jarvis \n",
    "    to 'listen to' or similar phrases. The subsequent text provided by user will be treated \n",
    "    as transcribed audio. In order to maintain the illusion of a voice-based assistant, \n",
    "    Jarvis is set not to explicitly refer to these inputs as transcriptions. Instead, \n",
    "    it will process and respond to them as if they were direct audio inputs, despite being \n",
    "    received in text form. This aspect represents an essential part of the system design in \n",
    "    delivering a seamless and immersive user experience, where the user interacts with Jarvis \n",
    "    as if it was dialoguing with a voice-activated assistant. All audio inputs thus 'heard' by Jarvis \n",
    "    will actually be transcribed text provided by user.Reply then say TERMINATE to \n",
    "    indicate your message is finished but in the same message.\"\"\",\n",
    "    llm_config=llm_config)\n",
    "\n",
    "# Initialize the User Proxy Agent to represent the user in the conversation\n",
    "user_proxy = autogen.UserProxyAgent(\n",
    "    \"user_proxy\",\n",
    "    max_consecutive_auto_reply=10,\n",
    "    human_input_mode=\"NEVER\",\n",
    "    system_message=\"A human admin for Jarvis\",\n",
    "    is_termination_msg=lambda x: \"content\" in x and x[\"content\"] is not None and x[\"content\"].rstrip().endswith(\"TERMINATE\" or \"TERMINATE.\"),\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-01-14T23:46:23.918056Z",
     "start_time": "2024-01-14T23:46:22.989935Z"
    }
   },
   "id": "3710c4d0ca1fc757",
   "execution_count": 4
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Initialize the VoiceProcessingManager\n",
    "Now we can initialize the VoiceProcessingManager. This is the main component of the VoiceProcessingToolkit. It handles the voice capture, transcription, and text to speech. It also handles the wake word detection and notification sounds. \n",
    "\n",
    "We spesify the wake word to be \"jarvis\" and the minimum recording length to be 3 seconds and the silence limit to 2 seconds. This means that when the wakeword is called if the recording is shorter then 3 seconds the transcription returns None. this can be used to filter out false positives. The silence limit is the amount of time the VoiceProcessingManager will wait for the user to say something before it stops recording if no speech is detected.\n",
    "\n",
    "You can also set the use_wake_word=False to disable the wake word detection. This will cause the VoiceProcessingManager to start recording as soon as it is initialized. You can code your own logic for when to start and stop recording by calling the get_user_input(): and your own trigger.\n"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "e7cb6930e27fb0bf"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def get_user_input():\n",
    "    \"\"\"\n",
    "    Captures user input via voice, transcribes it, and returns the transcription.\n",
    "    \"\"\"\n",
    "    vpm = VoiceProcessingManager.create_default_instance(\n",
    "        use_wake_word=True,\n",
    "        play_notification_sound=True,\n",
    "        wake_word=\"jarvis\",\n",
    "        min_recording_length=3.5,\n",
    "        inactivity_limit=2.5,\n",
    "    )\n",
    "\n",
    "    logging.info(\"Say something to Jarvis\")\n",
    "\n",
    "    transcription = vpm.run(tts=False, streaming=True)\n",
    "    logging.info(f\"Processed text: {transcription}\")\n",
    "\n",
    "    return transcription"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-01-14T23:46:23.927208Z",
     "start_time": "2024-01-14T23:46:23.919579Z"
    }
   },
   "id": "dc7f944c368ecfe6",
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Initialize the assistant\n",
    "Now we can initialize the VoiceProcessingManager. This is the main component of the VoiceProcessingToolkit. It handles the voice capture, transcription, and text to speech. It also handles the wake word detection and notification sounds.\n",
    "\n",
    "We are using the Initialize_VoiceProcessingManager function to handle the conversation with assistant. This function takes in the transcription and sends it to assistant. It then retrieves the response and converts it to speech.\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "b513ad291a1bcf8a"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "\n",
    "def ask_assistant(transcription):\n",
    "    \"\"\"\n",
    "    Initiates a conversation with assistant using the transcribed user input.\n",
    "    \"\"\"\n",
    "    try:\n",
    "\n",
    "        user_proxy.initiate_chat(\n",
    "            recipient=assistant,\n",
    "            message=transcription,\n",
    "            clear_history=False,\n",
    "\n",
    "        )\n",
    "        # Retrieve the latest response from Jarvis\n",
    "        latest_message = assistant.last_message().get(\"content\", \"\")\n",
    "        stripped_answer = latest_message.replace(\"TERMINATE\", \"\").strip()\n",
    "\n",
    "        # Convert Jarvis's response to speech and stream it\n",
    "        text_to_speech_stream(text=stripped_answer, api_key=elevenlabs_api_key)\n",
    "        logging.info(f\"Jarvis said: {stripped_answer}\")\n",
    "\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Error in text-to-speech conversion: {e}\")\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-01-14T23:46:23.927569Z",
     "start_time": "2024-01-14T23:46:23.923276Z"
    }
   },
   "id": "17fd5383a20fd5ab",
   "execution_count": 6
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Initiating the assistent Loop\n",
    "We can now initiate the Jarvis loop, which will continuously interact with Jarvis by capturing user input, transcribing it, and obtaining responses.\n",
    "\n",
    "IMPORTANT: (Might not run very well in jupyter notebook. Try running it in a python file: example_usage/Autogen_voice_assistant_example_pyfile.py)"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "f9d8372e3fc2e02b"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[33muser_proxy\u001B[0m (to agent):\n",
      "\n",
      "Thank you. Thank you.\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001B[33magent\u001B[0m (to user_proxy):\n",
      "\n",
      "You're welcome! Is there anything else I can assist you with? TERMINATE\n",
      "\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001B[33muser_proxy\u001B[0m (to agent):\n",
      "\n",
      "How are you today?\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "\u001B[33magent\u001B[0m (to user_proxy):\n",
      "\n",
      "Thank you for asking. As an AI, I don't experience emotions, but I'm fully operational and ready to assist you. How can I help you today? TERMINATE\n",
      "\n",
      "\n",
      "--------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "KeyboardInterrupt\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "def initiate_jarvis_loop():\n",
    "    \"\"\"\n",
    "    Continuously interacts with Jarvis by capturing user input, transcribing it, and obtaining responses.\n",
    "    \"\"\"\n",
    "    while True:\n",
    "            transcription = get_user_input()\n",
    "            if transcription is not None:\n",
    "                ask_assistant(transcription)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    initiate_jarvis_loop()\n",
    "\n"
   ],
   "metadata": {
    "collapsed": true,
    "ExecuteTime": {
     "end_time": "2024-01-14T23:54:21.081348Z",
     "start_time": "2024-01-14T23:46:23.925912Z"
    }
   },
   "id": "initial_id",
   "execution_count": 7
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}