• We use large language models (LLMs) to bridge natural language and behavior analysis.
• This work is published at NeurIPS2023! Read the paper, AmadeusGPT: a natural language interface for interactive animal behavioral analysis by Shaokai Ye, Jessy Lauer, Mu Zhou, Alexander Mathis & Mackenzie W. Mathis.
• Like this project? Please consider giving us a star ⭐️!

• AmadeusGPT is a Python package hosted on pypi. You can create a virtual env (conda, etc, see below*) and run:

pip install 'amadeusgpt[streamlit]'

Note that in order to access our demo video and keypoint files, we recommend to install from the source.

Make sure you edit those installation scripts to point to your own conda path before you run them

Recommended for: Running AmadeusGPT without GPUs. This setup is lightweight and is limited to processing movie files and keypoint outputs (.h5) from DeepLabCut.

# Install the minimal environment
bash install_minimal.sh

# Activate the conda environment
conda activate amadeusgpt-minimal

Recommended for: Users on Linux with GPUs. Support for Windows and MacOS will be added in the future.

# Install the gpu environment
bash install_gpu.sh

# Activate the conda environment
conda activate amadeusgpt-gpu

Recommended for: MacOS / Linux users working with very small video files.

# Install the cpu environment
bash install_cpu.sh

# Activate the conda environment
conda activate amadeusgpt-cpu

Why OpenAI API Key is needed AmadeusGPT relies on API calls of OpenAI (we will add more options in the future) for language understanding and code writing.

You can either add this into your environment by following:

export OPENAI_API_KEY='your API key' 

Or inside your python script or jupyter notebook, add this line in the beginning of the file

import os
os.environ["OPENAI_API_KEY"] = 'your api key' 

• Please note that you need an openAI API key, which you can easily create here.

make app

• If you only provide the raw video file, we use SuperAnimal models SuperAnimal models to predict your video. This is only supported with cpu or gpu installation. While we highly recommend gpu installation, we are working on faster, light-weight superanimal models to work on your CPU.
• If you already have keypoint file corresponding to the video file, look up how we set-up the config file in the notebooks. Right now we only support keypoint output from DeepLabCut. Other keypoint formats can be added upon feature requests.

We provide example notebooks at Notebooks

1. Draw ROI and ask when is the animal in the ROI. Demo
2. Obtain the binary mask for retrieved masks (for further sampling such as neural data pairing) Demo
3. Use SuperAnimal video inference (make sure you use gpu or cpu installation) if you don't have corresponding DeepLabCut keypoint file Demo
4. Write you own integration modules and use them Demo Source code. Make sure you delete the cached modules_embedding.pickle if you add new modules
5. Multi-Animal social interaction. Demo
6. Reuse the task program generated by LLM and run it on different videos Demo

If you use ideas or code from this project in your work, please cite us using the following BibTeX entry. 🙏

@article{ye2023amadeusGPT,
      title={AmadeusGPT: a natural language interface for interactive animal behavioral analysis}, 
      author={Shaokai Ye and Jessy Lauer and Mu Zhou and Alexander Mathis and Mackenzie Weygandt Mathis},
      journal={Thirty-seventh Conference on Neural Information Processing Systems},
      year={2023},
      url={https://openreview.net/forum?id=9AcG3Tsyoq},

• arXiv preprint version AmadeusGPT: a natural language interface for interactive animal behavioral analysis by Shaokai Ye, Jessy Lauer, Mu Zhou, Alexander Mathis & Mackenzie W. Mathis.

AmadeusGPT is license under the Apache-2.0 license.

• 🚨 Please note several key dependencies have their own licensing. Please carefully check the license information for DeepLabCut (LGPL-3.0 license), SAM (Apache-2.0 license), CEBRA (Non-Commercial), etc.

• 🤩 Dec 2023, code released!
• 🔥 Our work was accepted to NeuRIPS2023
• 🧙‍♀️ Open-source code coming in the fall of 2023
• 🔮 arXiv paper and demo released July 2023
• 🪄Contact us