Skip to content

moming133/shuziren_fay

 
 

Repository files navigation

虚拟人说话头生成(照片虚拟人实时驱动)

Get Started

Installation

Tested on Ubuntu 22.04, Pytorch 1.12 and CUDA 11.6,or Pytorch 1.12 and CUDA 11.3

git clone https://github.com/waityousea/xuniren.git
cd xuniren

Install dependency

# for ubuntu, portaudio is needed for pyaudio to work.
sudo apt install portaudio19-dev

pip install -r requirements.txt
or
## environment.yml中的pytorch使用的1.12和cuda 11.3
conda env create -f environment.yml 
## install pytorch3d
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

Build extension (optional)

By default, we use load to build the extension at runtime. However, this may be inconvenient sometimes. Therefore, we also provide the setup.py to build each extension:

# install all extension modules
bash scripts/install_ext.sh

start(独立运行)

环境配置完成后,启动虚拟人生成器:

python app.py

start(对接fay,在ubuntu 20下完成测试)

环境配置完成后,启动fay对接脚本

python fay_connect.py

扫码支助开源开发工作,凭支付单号入qq交流群

接口的输入与输出信息 Websoket.md

虚拟人生成的核心文件

## 注意,核心文件需要单独训练
.
├── data
│   ├── kf.json			
│   ├── pretrained
│   └── └── ngp_kg.pth

Inference Speed

在台式机RTX A4000或笔记本RTX 3080ti的显卡(显存16G)上进行视频推理时,1s可以推理35~43帧,假如1s视频25帧,则1s可推理约1.5s视频。

Acknowledgement

  • The data pre-processing part is adapted from AD-NeRF.
  • The NeRF framework is based on torch-ngp.
  • The algorithm core come from RAD-NeRF.
  • Usage example Fay.

学术交流可发邮件到邮箱:[email protected]

About

照片生产事实语言的数字人

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • HTML 91.3%
  • Python 5.9%
  • Cuda 2.6%
  • Other 0.2%