Generalizable Neural Voxels for Fast Human Radiance Fields
Taoran Yi^1*, Jiemin Fang^2,1*, Xinggang Wang^1✉, Wenyu Liu^1
1School of EIC, HUST   ²Institute of AI, HUST  
^* denotes equal contributions.

In this paper, we propose a rendering framework that can learn moving human body structures extremely quickly from a monocular video, named as GNeuVox. The framework is built by integrating both neural fields and neural voxels. Especially, a set of generalizable neural voxels are constructed. With pretrained on various human bodies, these general voxels represent a basic skeleton and can provide strong geometric priors. For the fine-tuning process, individual voxels are constructed for learning differential textures, complementary to general voxels. Thus learning a novel body can be further accelerated, taking only a few minutes. Our method shows significantly higher training efficiency compared with previous methods, while maintaining similar rendering quality.

Create and activate a virtual environment.

conda create --name gneuvox python=3.7
conda activate gneuvox

Install the required packages.

pip install -r requirements.txt

Download the gender neutral SMPL model from here, and unpack mpips_smplify_public_v2.zip.

Copy the smpl model.

SMPL_DIR=/path/to/smpl
MODEL_DIR=$SMPL_DIR/smplify_public/code/models
cp $MODEL_DIR/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl third_parties/smpl/models

Follow this page to remove Chumpy objects from the SMPL model.

First, download ZJU-Mocap dataset from here.

Second, modify the yaml file of subject 377 at tools/prepare_zju_mocap/377.yaml (e.g., subject 377).

Finally, run the data preprocessing script.

cd tools/prepare_zju_mocap
python prepare_dataset.py --cfg 377.yaml
python prepare_dataset_eval.py --cfg 377_eval.yaml
cd ../../

First, you have to first read and agree the license terms here. Then you can download the data we have processed with EasyMocap from google drive and baidu drive.

Use EasyMocap to process the video and run the data preprocessing script.

cd tools/prepare_easy_mocap
python prepare_datase_easy_mocap.py --cfg easy_mocap.yaml
cd ../../

python train.py --cfg configs/scene/zju_mocap/377.yaml

You can download the pretrained models from google drive and baidu drive. Put the pretrained models in experiments/$task/$experiment. Then use _C.resume = True in configs/config.py to load the pretrained models.

You can use the following script to evaluate the model.

python eval.py --cfg configs/scene/zju_mocap_eval/377.yaml 

Render the frame input (i.e., training sequence).

python run.py --type movement \
   --cfg configs/scene/zju_mocap_eval/377.yaml   

Run free-viewpoint rendering on a particular frame (e.g., frame 128).

python run.py --type freeview \
    --cfg configs/scene/zju_mocap_eval/377.yaml   \
    freeview.frame_idx 128

Render the learned canonical appearance (T-pose).

python run.py --type tpose \
    --cfg configs/scene/zju_mocap_eval/377.yaml   

Our code is partially based on HumanNeRF and DirectVoxGO. Thanks for their awesome works.

If you find this repository/work helpful in your research, welcome to cite the paper and give a ⭐.

@article{gneuvox,
        title={Generalizable Neural Voxels for Fast Human Radiance Fields},
        author={Taoran Yi and Jiemin Fang and Xinggang Wang and Wenyu Liu},
        journal={arxiv:2303.15387},
        year={2023}
        }