Skip to content

Code for "Representing Volumetric Videos as Dynamic MLP Maps" CVPR 2023

License

Notifications You must be signed in to change notification settings

zju3dv/mlp_maps

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Representing Volumetric Videos as Dynamic MLP Maps

teaser

Representing Volumetric Videos as Dynamic MLP Maps
Sida Peng*, Yunzhi Yan*, Qing Shuai, Hujun Bao, Xiaowei Zhou (* equal contribution)
CVPR 2023

Any questions or discussions are welcomed!

Installation

Please see INSTALL.md for manual installation.

Interactive demo

Interactive rendering on ZJU-MoCap

Please see INSTALL.md to download the dataset. We provide the pretrained models at here.

Take the rendering on sequence 313 as an example.

  1. Download the corresponding pretrained model and put it to $ROOT/data/trained_model/zjumocap/313/final.pth.

  2. Interactive rendering demo:

    python gui.py --config configs/zjumocap/dymap_313.py fast_render True 
    
Interactive rendering on NHR

Please see INSTALL.md to download the dataset. We provide the pretrained models at here.

Take the rendering on sequence sport1 as an example.

  1. Download the corresponding pretrained model and put it to $ROOT/data/trained_model/nhr/sport1/final.pth.

  2. Interactive rendering demo:

    python gui.py --config configs/nhr/sport1.py fast_render True 
    

Run the code on ZJU-MoCap

Please see INSTALL.md to download the dataset.

We provide the pretrained models at here.

Test on ZJU-MoCap

Take the test on sequence 313 as an example.

  1. Download the corresponding pretrained model and put it to $ROOT/data/trained_model/zjumocap/313/final.pth.

  2. Test on unseen views:

    python run.py --config configs/zjumocap/dymap_313.py mode evaluate fast_render True
    
Visualization on ZJU-MoCap

Take the visualization on sequence 313 as an example.

  1. Download the corresponding pretrained model and put it to $ROOT/data/trained_model/zjumocap/313.

  2. Visualization:

    • Visualize free-viewpoint videos
    python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fast_render True
    

    free-viewpoint video

    • Visualize novel views of single frame
    python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fixed_time True fast_render True
    

    novel_view

    • Visualize the dynamic scene with fixed camera
    python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fixed_view True fast_render True
    

    time

    • Visualize mesh
    python run.py --config configs/zjumocap/dymap_313.py  mode visualize vis_mesh True  fast_render True    
    
Training on ZJU-MoCap

Take the training on sequence 313 as an example.

  1. Train:
    # training
    python train_net.py --config configs/zjumocap/dymap_313.py
    # distributed training
    python -m torch.distributed.launch --nproc_per_node=4 train_net.py --config configs/zjumocap/dymap_313.py
    
  2. Post-process the trained model:
    python run.py --config configs/zjumocap/dymap_313.py mode visualize occ_grid True
    
  3. Tensorboard:
    tensorboard --logdir data/record/zjumocap
    

Run the code on NHR

Please see INSTALL.md to download the dataset.

We provide the pretrained models at here.

Test on NHR

Take the test on sequence sport1 as an example.

  1. Download the corresponding pretrained model and put it to $ROOT/data/trained_model/nhr/sport1/final.pth.

  2. Test on unseen views:

    python run.py --config configs/nhr/sport1.py mode evaluate fast_render True
    
Visualization on NHR

Take the visualization on sequence sport1 as an example.

  1. Download the corresponding pretrained model and put it to $ROOT/data/trained_model/nhr/sport1.

  2. Visualization:

    • Visualize novel views
    python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fast_render True
    

    free-viewpoint video

    • Visualize novel views of single frame
    python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fixed_time True fast_render True
    

    novel_view

    • Visualize the dynamic scene with fixed camera
    python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fixed_view True fast_render True
    

    time

    • Visualize mesh
    python run.py --config configs/nhr/sport1.py mode visualize vis_mesh True  fast_render True    
    
Training on NHR

Take the training on sequence sport1 as an example.

  1. Train:
    # training
    python train_net.py --config configs/nhr/sport1.py
    # distributed training
    python -m torch.distributed.launch --nproc_per_node=4 train_net.py --config configs/nhr/sport1.py
    
  2. Post-process the trained model:
    python run.py --config configs/nhr/sport1.py mode visualize occ_grid True
    
  3. Tensorboard:
    tensorboard --logdir data/record/nhr
    

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{peng2023representing,
  title={Representing Volumetric Videos as Dynamic MLP Maps},
  author={Peng, Sida and Yan, Yunzhi and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
  booktitle={CVPR},
  year={2023}
}