Official Repository for CVPR 2024 paper SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting.
The embedding points of 3DGS on the triangle mesh are updated by the walking on triangle scheme.
See the phongsurface module implemented in c++ and pybind11.
- Create conda env with pytorch.
conda create -n splatting python=3.9
conda activate splatting
# pytorch 1.13.1+cu117 is tested
pip install torch==1.13.1 torchvision torchaudio functorch --extra-index-url https://download.pytorch.org/whl/cu117
# pytorch3d
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
pip install -e .
# install other dependencies
pip install tqdm omegaconf opencv-python libigl
pip install trimesh plyfile imageio chumpy lpips
pip install packaging pybind11
pip install numpy==1.23.1
- Clone this repo recursively. Install Gaussian Splatting's submodules.
git clone --recursive https://github.com/initialneil/SplattingAvatar.git
cd SplattingAvatar
cd submodules/diff-gaussian-rasterization
pip install .
cd ../submodules/simple-knn
pip install .
cd ../submodules/multi-hmr
pip install -r requirements.txt
cd ..
- Install
simple_phongsurffor walking on triangles.
cd model/simple_phongsurf
pip install -e .
-
Download FLAME model, choose FLAME 2020 and unzip it, copy 'generic_model.pkl' into
./model/imavatar/FLAME2020, -
Download SMPL model (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding places:
mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl model/smplx_utils/smplx_models/smpl/SMPL_FEMALE.pkl
mv /path/to/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl model/smplx_utils/smplx_models/smpl/SMPL_MALE.pkl
We provide the preprocessed data of the 10 subjects used in the paper.
- Our preprocessing followed IMavatar and replaced the Segmentation with RobustVideoMatting.
- Pre-trained checkpoints are provided together with the data.
- Google Drive or Baidu Disk
We conducted experiments on PeopleSnapshot.
- Please download the parameter files (the same with InstantAvatar) from: Baidu Disk or Google Drive.
- Download 4 sequences from PeopleSnapshot (male/female-3/4-casual) and unzip
imagesandmasksto corresponding folders from above. - Use
scripts/preprocess_PeopleSnapshot.pyto preprocess the data. - Pipeline:
Run multi-hmr submodule to estimate SMPLX parameters for input images
python inference.py --img_folder /path/to/male-3-casual/images --out_folder /path/to/outdir --model_name multiHMR_896_L
Preprocess to get mask of object
python scripts/preprocess_PeopleSnapshot.py --root path/to/peoplesnapshot/ --outdir /path/to/male-3-casual/
Training
python train_splatting_avatar.py --config "configs/splatting_avatar_smplx.yaml;configs/instant_avatar_smplx.yaml" --dat_dir /path/to/male-3-casual/
Inference on new pose
python eval_animate_smplx.py --config "configs/splatting_avatar_smplx.yaml;configs/instant_avatar_smplx.yaml" --dat_dir /path/to/male-3-casual/ --pc_dir /path/to/male-3-casual//output-splatting/<exp_name>/point_cloud/iteration_30000/ --anim_fn assets/pose.pkl
We conducted our experiments on a single NVIDIA RTX 3090 with 24GB. Training with less GPU memory can be achieved by setting a maximum number of Gaussians.
# in configs/splatting_avatar.yaml
model:
max_n_gauss: 300000 # or less as needed
or set by command line
python train_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir <path/to/subject> model.max_n_gauss=300000
If you find our code or paper useful, please cite as:
@inproceedings{SplattingAvatar:CVPR2024,
title = {{SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting}},
author = {Shao, Zhijing and Wang, Zhaolong and Li, Zhuang and Wang, Duotun and Lin, Xiangru and Zhang, Yu and Fan, Mingming and Wang, Zeyu},
booktitle = {Computer Vision and Pattern Recognition (CVPR)},
year = {2024}
}
We thank the following authors for their excellent works!
SplattingAvatar
The code is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License for Noncommercial use only. Any commercial use should get formal permission first.
Gaussian Splatting
Inria and the Max Planck Institut for Informatik (MPII) hold all the ownership rights on the Software named gaussian-splatting. The Software is in the process of being registered with the Agence pour la Protection des Programmes (APP).