Dolphin is a general video interaction platform based on large language models. Our team is trying to build a chatbot for video understanding, processing and generation.
The project 🐬 Dolphin is still under construction, we will continue to update it and welcome contributions/pull requests from the community.
Stable online demo is coming soon! See About in Github for the temporary demo URL.
![Dolphin, a general video interaction platform based on LLMs, from BUAA & NTU](https://user-images.githubusercontent.com/78398294/236398299-050439e4-b870-44f5-8341-db87ca849748.png)
-
2023/05/06: Code release & Online Demo
- Video understanding: Q&A about the video.
- Video processing: Basic functions such as trimming video, adding subtitles, extracting audio, and adding audio using moviepy. Video to pose/depth/canny also included.
- Video generation: Text to video, pose/depth and text to video, and video pix2pix.
example_222_2x.mp4
Prepare the project and environment:
# We recommend using conda to manage the environment and use python 3.8
conda create -n dolphin python=3.8
conda activate dolphin
# Clone the respository:
git clone https://github.com/BUAA-PrismGroup/dolphin.git
cd dolphin
# Install dependencies:
pip install -r requirements.txt
To start 🐬 Dolphin, you can specify the GPU/CPU assignment by --load
, the parameter indicates which Video Foundation Model to use and where it will be loaded to. The model and device are separated by underline _
, while the different models are separated by comma ,
. The available Video Foundation Models can be found in the following table or configs/backends.yaml
.
For example, if you want to load VideoCaptioning to cuda:0 and MoviepyInterface to cpu, you can use: VideoCaptioning_cuda:0,MoviepyInterface_cpu
.
Some starting commands are as follows.
# Advice for CPU Users
python video_chatgpt.py --load VideoCaptioning_cpu,ImageCaptioning_cpu,ModelscopeT2V_cpu
# Advice for 4 Tesla V100 32GB (Full usage)
# You can specify the device where each model is loaded in `configs/backend.yaml`
python video_chatgpt.py
Foundation Model | GPU Memory (MB) |
---|---|
VideoCaptioning | 13393 |
ImageCaptioning | 8429 |
MoviepyInterface | 0 |
Video2Canny | 0 |
Video2Pose | 1361 |
Video2Depth | 1521 |
CannyText2Video | 6659 |
PoseText2Video | 6721 |
DepthText2Video | 6673 |
VideoPix2Pix | 5251 |
ModelscopeT2V | 6535 |
Text2Audio | 5797 |
Our project framework is highly extensible for adding new features, including support for more video foundation models and more large language models.
For more video foundation models, you can add the inference code for new models under the modules
directory. We recommend creating a new Python package for the new model within this directory and implementing the class in the package's __init__.py
file (you can refer to ModelscopeT2V
in modules/modelscope_t2v/init.py
). Afterward, add the related information in configs/backends.yaml
.
Example: FaceText2Video
Assuming you have now implemented a new video foundation model using facial keypoints and text descriptions to generate videos, follow these steps:
- Create a new package named
face2video
under the directorymodules
. In its__init__.py
file, implement a class calledFaceText2Video
, which includes an initialization and an inference function. The desired effect should be that by importingmodules.face2video.FaceText2Video
, after instantiating an object, you can use theinference
function to achieve the corresponding functionality. - Add the import and function description in
configs/backends.yaml
. Details are as follows.
model_zoos:
FaceText2Video: # foundation model class
target: modules.face2video.FaceText2Video # path of the class in project
params: # params passed to the class
device: cuda:0
tools:
# - name: tool name
# desc: description about new tool
# instance: keep the name consistent with the one in the model_zoos section mentioned above
# func: inference function in foundation model class
- name: Generate Video Condition On Face Video
desc: "useful when you want to generate a new video from both the user description and a facial keypoints video. /
like: generate a new video of a human face from this human face video, or can you generate a video based on both the text 'A boy is playing basketball.' and this face video. /
The input to this tool should be a comma separated string of two, representing the video_path and the user description. "
instance: FaceText2Video
func: inference
For more large language models, you can refer to video_chatgpt.py
, and create a new file like video_moss.py
or video_stablelm.py
in the project root directory to implement support for other large language models.
- Chatbot with video downstream works (video understanding, processing and generation)
- Pretrained unified video model with in-context learning
- Benchmark for emerging video tasks
- Service including Gradio, Web, API and Docker
We appreciate the open source of the following projects:
Hugging Face LangChain mPLUG BLIP-2 MoviePy Text2Video-Zero damo/text-to-video-synthesis bark
For help or issues using the 🐬 Dolphin, please submit a GitHub issue.
For other communications, please contact Zehuan Huang ([email protected]) or kaleido lab ([email protected]). Welcome to follow us in Twitter (@kaleido_lab).
If you find this repository useful, please consider citing:
@misc{dolphin,
Author = {Zehuan Huang, Haoran Feng, Chongzhi Zhang, Lu Sheng, Ziwei Liu, Jing Shao},
Year = {2023},
Note = {https://github.com/kaleido-lab/dolphin},
Title = {Dolphin: General Video Interaction Platform Based on LLMs}
}