Yiming Wang*, Qin Han*, Marc Habermann, Kostas Daniilidis, Christian Theobalt, Lingjie Liu
ICCV 2023
NeuS2 is a method for fast neural surface reconstruction, which achieves two orders of magnitude improvement in terms of acceleration without compromising reconstruction quality, compared to NeuS. To accelerate the training process, we integrate multi-resolution hash encodings into a neural surface representation and implement our whole algorithm in CUDA. In addition, we extend our method for reconstructing dynamic scenes with an incremental training strategy.
This project is an extension of Instant-NGP enabling it to model neural surface representation and dynmaic scenes. We extended:
- dependencies/neus2_TCNN
- add second-order derivative backpropagation computation for MLP;
- add progressive training for Grid Encoding.
- neural-graphics-primitives
- extend NeRF mode for NeuS;
- add support for dynamic scenes.
-
[08/30/2023] Released NeuS2++ (See neuspp branch). Now you can reconstruct unmasked/unbounded scenes in seconds!
-
[08/15/2023] Released official codes!
Synthetic Scene surface reconstruction in 5 minutes Input: multi-view images with mask Output: freeview synthesis, mesh static-scene-lego-5min.mp4 |
Synthetic Scene surface reconstruction in 5 minutes Comparison between NeuS (8h), Instant-NGP (5min) and NeuS2 (5min) Download dataset from Google Drive static-scene-dtu-scan20-compressed.mov |
Dynamic Scene surface reconstruction in 20s per frame Input: a sequence of multi-view images with mask Output: novel view synthesis, mesh dynamic-scene-dynacap.mp4 |
Long sequence surface reconstruction with 2000 frames Input: a sequence of multi-view images with mask NeuS2 can handle long sequences input with large movements long_sequences_compressed.mp4 |
Please first see Instant-NGP for original requirements and compilation instructions. NeuS2 follows the installing steps of Instant-NGP.
Clone this repository and all its submodules using the following command:
git clone --recursive https://github.com/19reborn/NeuS2
cd NeuS2
Then use CMake to build the project:
cmake . -B build
cmake --build build --config RelWithDebInfo -j
For python useage, first install dependencies with conda and pip:
conda create -n neus2 python=3.9
conda activate neus2
pip install -r requirements.txt
Then install pytorch and pytorch3d.
If you meet problems of compiling, you may find solutions here.
You can specify a static scene by setting --scene
to a .json
file containing data descriptions.
The DTU Scan24 scene can be downloaded from Google Drive:
./build/testbed --scene ${data_path}/transform.json
Or, you can run the experiment in an automated fashion through python bindings:
python scripts/run.py --scene ${data_path}/transform.json --name ${your_experiment_name} --network ${config_name} --n_steps ${training_steps}
For training on DTU dataset, config_name
can be dtu.json
and training_steps
can be 15000
.
Also, you can use scripts/run_dynamic.py
as:
python scripts/run_dynamic.py --scene ${data_path}/transform.json --name ${your_experiment_name} --network ${config_name}
, where the number of training iterations is specified in the config.
The outputs and logs of the experiment can be found at output/${your_experiment_name}/
.
To specify a dynamic scene, you should set --scene
to a directory containing .json
files that describe training frames.
./build/testbed --scene ${data_dirname}
Or, run scripts/run_dynamic.py
using python:
python scripts/run_dynamic.py --scene ${data_dirname} --name ${your_experiment_name} --network ${config_name}
There are some hyperparameters of the network configuration, such as configs/nerf/base.json
, to control the dynamic training process:
first_frame_max_training_step
: determine the number of training iterations for the first frame, default2000
.next_frame_max_training_step
: determine the number of training iterations for subsequent frames, default1300
, including global transformation prediction.predict_global_movement
: settrue
if use global transformation prediction.predict_global_movement_training_step
: determine the number of training iterations for global transformation prediction, default300
. Only valid whenpredict_global_movement
istrue
.
Also, we provide scripts to reconstruct dynamic scenes by reconstructing static scene frame by frame.
python scripts/run_per_frame.py --base_dir ${data_dirname} --output_dir ${output_path} --config ${config_name}
Dynamic scene examples can be downloaded from Google Drive.
- Static scene example, link.
- Dynamic scene example, link.
- Pretrained model and configuration files for DTU, link.
NeuS2 supports the data format provided by Instant-NGP. Also, you can use NeuS2's data format (with from_na=true
), see data convention.
We also provide a data conversion from NeuS to our data convention, which can be found in tools/data_format_from_neus.py
.
@inproceedings{neus2,
title={NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction},
author={Wang, Yiming and Han, Qin and Habermann, Marc and Daniilidis, Kostas and Theobalt, Christian and Liu, Lingjie},
year={2023},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}
}
@article{mueller2022instant,
author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
journal = {ACM Trans. Graph.},
issue_date = {July 2022},
volume = {41},
number = {4},
month = jul,
year = {2022},
pages = {102:1--102:15},
articleno = {102},
numpages = {15},
url = {https://doi.org/10.1145/3528223.3530127},
doi = {10.1145/3528223.3530127},
publisher = {ACM},
address = {New York, NY, USA},
}
@inproceedings{wang2021neus,
title={NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction},
author={Wang, Peng and Liu, Lingjie and Liu, Yuan and Theobalt, Christian and Komura, Taku and Wang, Wenping},
booktitle={Proc. Advances in Neural Information Processing Systems (NeurIPS)},
volume={34},
pages={27171--27183},
year={2021}
}