Jarvis is the built-in AI agent/AI butler residing on OpenDAN AI OS. It's a prototype of an AI assistant that is intelligent and extensible.
On the surface, Jarvis may appear similar to those existing AI assistants like Cortana or Siri, but behind the scenes it is quite different. Jarvis's intelligence is powered by large language models, giving it a degree of thinking and reasoning abilities. The LLM engine of Jarvis can be configured to closed-source products like ChatGPT or Claude, as well as open-source LLMs like Vicuna, Alpaca and so on.
Also, unlike traditional AI assistants, Jarvis is customizable and extensible. You can freely and easily provide Jarvis with more skills by installing functional modules and services to the OpenDAN AI OS, enabling it to do more for you. While it starts as an AI assistant, helping with basic tasks like scheduling meetings or controlling smart home devices, Jarvis can grow into so much more. We built Jarvis to be an open-ended platform for artificial intelligence at home. With consistent improvements over time, Jarvis may become far more advanced than any existing consumer AI product.
First we need to config LLM engine. At present, we use ChatGPT as the LLM engine and you need to provide an OpenAI API KEY in the .env
config file.
Follow the code below:
# Assuming your current workding directory is ${project_root}, i.e., the directory containing this README.md file.
# Pull the docker image
# NOTE: You may need root privilege, if your docker daemon only allows root.
docker pull fiatrete/jarvis
# Create our workspace and enter
mkdir -p docker/jarvis
cd docker
# copy necessary files
cp ../.env_template jarvis/.env
# Edit '.env' and configure JARVIS_OPENAI_API_KEY.
# By default Jarvis uses gpt-3.5. You can use other version by setting JARVIS_LLM_MODEL and JARVIS_SMALL_LLM_MODEL.
vim jarvis/.env
# Assuming jarvis's listening port is 10000 (According to you '.env' configuration).
# WARNING for windows users:
# I don't know why, but as a fact, executing the command below using git-bash may fail due to docker maps a wrong path.
# You can try to use powershell/cmd to run the docker container. Don't forget to change '`pwd`' to the absolute path of
# our current workding directory, i.e., the 'docker' folder.
docker run --name jarvis --rm \
-p 10000:10000 \
-v `pwd`/jarvis/.env:/root/.env \
fiatrete/jarvis:latest
Then you can open your browser, and navigate to http:https://127.0.0.1:10000
.
You will see a simple chat window, and you can chat with Jarvis with it. This is a showcase of the interaction with Jarvis. It is more practical to use Telegram or discord bot to interact with Jarvis and we will provide the code later.
At this moment, Jarvis is most likely a copy of ChatGPT, except that we provide some simple abilities such as telling joke and removing images background as shown above.
To enable Jarvis with drawing capabilities, you will need to set up an Stable Diffusion webui with --api
enabled. The OpenDAN AI OS will provide a one-click installation of Stable Diffusion webui in the future, but you will have to install it by yourself for now.
# Assuming your current workding directory is ${project_root}, i.e., the directory containing this README.md file.
cd docker
# Edit '.env' and configure DEMO_STABLE_DIFFUSION_ADDRESS
# e.g. http:https://192.168.3.254:1997
# NOTE: Do *** NOT **** use 'localhost' or '127.0.0.1' or other loopback address,
# since we don't use host network in docker.
vim jarvis/.env
# Assuming jarvis's listening port is 10000 (According to you '.env' configuration).
docker run --name jarvis --rm \
-p 10000:10000 \
-v `pwd`/jarvis/.env:/root/.env \
jarvis:latest
Then you can ask Jarvis to generate images for you in the chat window. You may say "Jarvis, generate a photo of a beautiful girl" for example.
To enable Jarvis with the ability to summarize YouTube videos and send tweets you need to:
- provide Twitter API keys and setup them in
.env
. see Appendix A.1
# == Twitter
# https://developer.twitter.com/en/portal/dashboard
# The consumer key
DEMO_TWITTER_CONSUMER_KEY=
DEMO_TWITTER_CONSUMER_SECRET=
# The access token (not 'Bearer Token')
DEMO_TWITTER_ACCESS_TOKEN=
DEMO_TWITTER_ACCESS_TOKEN_SECRET=
# Your twitter account, i.e., the last part of your twitter profile page, the part followed by '@'
# e.g. Elon Musk, his account names is 'elonmusk'
DEMO_TWITTER_USERNAME=
- provide YouTube access key and setup
DEMO_YOUTUBE_API_KEY
in.env
. see Appendix A.2
Then startup Jarvis open the chat window in your browser:
cd docker
# Assuming jarvis's listening port is 10000 (According to you '.env' configuration).
docker run --name jarvis --rm \
-p 10000:10000 \
-v `pwd`/jarvis/.env:/root/.env \
jarvis:latest
Now you can send YouTube urls to Jarvis and ask him to summarize the content of the video. Also, you can ask him to send tweets.
There are some example external functional modules in ${project_root}/../example_modules
directory.
The main program of Jarvis will scan JARVIS_EXTERNAL_FUNCTION_MODULE_DIR
recursively, and load all functional modules found.
A functional module is denoted by a .module.py
file. Once a .module.py
file found, then this directory is a functional module, then .module.py
will be loaded as a python module, and the subdirectories of this module will not be further scaned.
To make the life easier, the funcitonal module directories will be added to python's module search pathes. Thus, you can import python modules using a package name relative to the .module.py
file's containing directory. And also, watch out module name confliction.
Please refer to the example modules, and try to write a module by yourself.
If you cannot run Jarvis successfully, you can set JARVIS_LOG_LEVEL
to debug
, and set JARVIS_DEBUG_MODE
to true
, in the .env
file. And run it again.
The log file is located in the docker container at:
/root/jarvis/log.txt
: This is the main log./root/jarvis/supervisor_log/{xxx}_std{yyy}.log
: These arestderr
andstdout
of our services.
Q: Docker repeatedly says exit: {some service} (exit status {y}, not expected)
A: Check the .env
file.
- Ensure that the
.env
file is mapped correctly. - Ensure that all mandatory options in the
.env
file are set.
Q: Jarvis always reply: Sorry, but I don't know what you want me to do.
A: Check the .env
file.
- Ensure that
JARVIS_OPENAI_API_KEY
is set correctly; - Ensure that
JARVIS_OPENAI_API_KEY
is valid; - Ensure that you can access OpenAI' API;
- You may need to set
JARVIS_OPENAI_URL_BASE
if your key is not provied by OpenAI;
Q: Jarvis always fails to execute twitter
/youtube
/stable-diffusion
related job.
A: Check the .env
file. Ensure that the corresponding configurations are all set correctly.
If your issue is not listed above or you have any questions. Feel free to open an issue. We are happy to discuss with you.
- Register a twitter developer account.
- Create an application in twitter portal dashboard.
- In your app detail page, navigate to
Settings -> User authentication settings -> Edit
. - Select
Read and write
inApp permissions
table. - Navigate to
Keys and Tokens
tab. - Create an
API Key and Secret(Consumer Keys)
and anAccess Token and Secret
.
NOTE: You must select Read and write
first, then create keys. If the keys were created first, you will have to recreate the keys again.
Both these keys and their corresponding secrets are required in the .env
file.
This is relatively simple.
- Register a google developer account. (You might have one already.)
- Create a project in google api dashboard, and select it.
- Enable
YouTube Data API v3
. - Create
API Key
at google credentials page. If you were asked to configure an OAuth page, do it first.
The created API key string is what we want.
- Register a google developer account.
- Create a project in google api dashboard, and select it.
- Enable
Google Calendar API
. - Create
OAuth client ID
at google credentials page. If you were asked to configure an OAuth page, do it first. a. SelectDesktop Application
. - After
OAuth ID
created, download it by clicking theDowload JSON
button. - Place the downloaded file as
${project_root}/../example_services/demo_service2/credentials.json
- You'll have to run
demo_service2
first, and visit itsdocs
page (http:https://127.0.0.1:8000/docs, change the port to your actual listening port) - At this page, expand
/tasks
, clicktry it out
, then clickExecute
. - You browser will be invoked to a google's auth page. You are asked to authorize our
demo_service2
to access your google account. Just accept. - The file
token.json
will be automatically stored at${project_root}/../example_services/demo_service2/token.json
See also https://developers.google.cn/calendar/api/quickstart/python.