Skip to content

Real-time video understanding and interaction through text,audio,image and video with large multi-modal model. 利用多模态大模型的实时视频理解和交互框架,通过文本、语音、图像和视频和这是世界进行问答和交流。

License

Notifications You must be signed in to change notification settings

IDEA-CCNL/Real-Gemini

Repository files navigation

Real-Gemini

Real-time video understanding and interaction through text, audio, image and video with large multi-modal model.

利用多模态大模型的实时视频理解和交互框架,通过文本、语音、图像和视频和这是世界进行问答和交流。

环境配置

首先需要根据requirements.txt安装相关的python包。

pip install -r requirements.txt

在前端环境,同时要安装pyaudio。

sh install pyaudio.sh

你需要设置一些环境变量。首先,将.env.template复制到同目录下的.env

cp .env.template .env

然后填写对应的环境变量。其中,请注意以下的环境变量:

启动后端模型服务

你可以在有GPU的后端环境中启动服务,例如启动TTS的服务。

sh scripts/start_tts_server.sh

必须要启动的服务有:

  • TTS

可选启动的服务有:

  • Music:当你需要生成音乐时;
  • Qwen:当你需要使用Qwen-VL替换GPT-4V时;

启动前端对话服务

主要实现使用streamlit实现了前端界面。

然后启动前端:

sh run.sh

请注意,若前端环境是MacOS,请不要使用第三方的Terminial启动服务。

ASR服务

  • ASR 服务调用自内部自建工具(TODO:更新一个服务并开源)

Acknowledgement

关于我们 About Us

IDEA研究院封神榜团队是中文大模型开源计划Fengshenbang-LM的负责团队,开源包括二郎神系列太乙系列姜子牙系列等知名模型,并收获了开源社区的广泛使用和支持。

IDEA研究院CCNL技术团队已创建封神榜开源讨论群,我们将在讨论群中不定期更新发布封神榜新模型与系列文章。请扫描微信搜索“fengshenbang-lm”,添加封神空间小助手进群交流!

The IDEA Research Institute Fengshenbang team is the responsible team for the Chinese large model open source project Fengshenbang-LM. The open source includes well-known models such as the Erlang, Taiyi, and Ziya, and has received widespread use and support from the open source community.

The IDEA Research Institute CCNL technical team has created an open discussion group for Fengshenbang. We will periodically update and release new Fengshenbang models and series of articles in the discussion group. Please scan WeChat and search for "fengshenbang-lm", and add the Fengshen Space Assistant to join the group discussion!

About

Real-time video understanding and interaction through text,audio,image and video with large multi-modal model. 利用多模态大模型的实时视频理解和交互框架,通过文本、语音、图像和视频和这是世界进行问答和交流。

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published