@article{Jaey2023DNGP,
author = {Choi, Jaeyeol and Jeong, Jong-Beom and Park, JunHyeong and Ryu, Eun-Seok},
title = {A Deep Learning-based 6DoF Video Synthesizing Method Using Instant-NGPs},
booktitle = {2023 IEEE Visual Communications and Image Processing (VCIP)},
year = {2023},
}
Represent 3D dynamic data implicitly on Neural Network models, and render the video based on your own pose trace.
This software is implemented based on instant-NGP and camorph.
By training "instant-NGP" model using transfer learning per frame, We provide temporal-consistent video with relatively-high-speed.
For camera parameter and pose trace format, we adopted MPEG-Immersive-Video(MIV) standard.
- Input: Set of
.yuv
files, MIV-format camera parameter file, and (optionally) MIV-format pose trace file. (depth map is NOT required.) - Output: Rendered video of novel views containing dynamic scene.
The overall time of data transforming, training, and rendering is about 1 hour for 90 frames.
- All process shown above are implemented as two scripts,
train.py
andrender.py
. - You train the dynamic scene with
python train.py
. The conditions you need to give (e.g. location of video files, or iterations settings) should be entered ontrain_config.json
- After you train the models, you can render the video anytime you want by running
python render.py
. The conditions you need to give (e.g. which models you will use, or the pose trace you want to render) should be entered ontest_config.json
Some features are not implemented yet (e.g. using colmap). It will be updated soon.
- The concrete explaination is written below.
For now, this software is available only on Linux or WSL2. (not native Windows)
1. Pre requirements
- CUDA 10.2 or higher
- click here for CUDA installation for linux
- click here for CUDA installation for WSL2.
- in
~/.bashrc
or~/.zshrc
, those line should be added. (e.g. if you have CUDA 11.7). Then runsource ~/.bashrc
orsource ~/.zshrc
.
export PATH="/usr/local/cuda-11.7/bin:$PATH" export LD_LIBRARY_PATH="/usr/local/cuda-11.7/lib64:$LD_LIBRARY_PATH"
- Anaconda
- click here for anaconda installation.
- run
conda --version
to check whether it's available.
- CMake v3.21 or higher
- click here for download.
2. Cloning the repository
git clone --recursive https://github.com/jaeyeol816/Dynamic-NGP.git
cd Dynamic-NGP
- WARNING: Don't forget to include
--recursive
option when cloning.
3. Create and activate anaconda environment
conda env create -f dy_ngp.yml
- The command above helps you make the suitable anaconda environment named
dy_ngp
for this software.
conda activate dy_ngp
- Activate the
dy_ngp
environment.
4. Installing the packages
sudo apt-get install ffmpeg build-essential git python3-dev python3-pip libopenexr-dev libxi-dev libglfw3-dev libglew-dev libomp-dev libxinerama-dev libxcursor-dev
5. Compliation
cmake . -B build
cmake --build build --config RelWithDebInfo -j
- If you are running out of memory, try running the command without
-j
. - The command would take some time.
- If the building error occured, the issues from instant-ngp will help.
- Although, the building step for this sortware is not identical to instant-ngp. The
CMakeLists.txt
in this project set to do not build GUI part of instant-ngp.
- Although, the building step for this sortware is not identical to instant-ngp. The
In constrast to building step, the running step is quite simple.
To summerize, the only thing you have to do is (1) filling the train_config.json
and (2) runnning python script as python train.py
.
1. Fill config file
Fill out the train_config.json
file to give information such as location of the yuv files and iteration time.
The term train_id
means a "training experiment". The multiple "rendering experiment" can be done with one training experiment
train_exp_id
: Any name can be set to represent specific traininig experiment.path_of_dir_containing_only_texture_yuv
: The path of Folder that contains bunch of yuv files.- WARNING: depth file should NOT be in this directory.
- WARNING: The file name should contain string that show view info as
v2_
orv02_
. Then the software will automatically detect your view number.
path_of_MIV_json_file
: the location of camera path json file likeA.json
,S.json
image_base_dir
: The location where we will store the convertedpng
files. If they already exist, we will not convert.initial_n_iters
: The number of iterations of the first frame.transfer_learning_n_iters
: The number of iterations of the second~last frame. For specific frame, the training is held using weights(parameter) of previous frame.frame_start
,frame_end
: You can set how much frame you will train.exclude_specific_views
: if "true", then we will not train Instant-NGP with the views which you designate inviews_to_exclude
2. Run python file
python train.py
- All the step will be done automatically.
- It will take some time (depending on
transfer_learning_n_iters
and num of frames). Take some coffee, or have dinner.
3. Location of models and log files
- The location of the models:
{result_dir}/train_{train_id}/render_{render_id}/models
- The location of the log file:
{result_dir}/train_{train_id}/log.txt
- the time info and training info are written in log file
- you should not delete model and log file, because it will be used in rendering process.
The rendering experiment is subordinated to training experiemnt.
To summerize, the only thing you have to do is (1) filling the render_config.json
and (2) runnning python script as python render.py
.
1. Fill config file
-
train_exp_id
: Specifying the trained model you want to render. -
render_exp_id
: Any name that represents this rendering experiment. The many rendering experiemnt is available on one training experiment. -
result_dir
: It should be same asresult_dir
of your training case (the training experiment thattrain_exp_id
points) -
frame_start
,frame_end
: The frames you want to render. It should be subset of the frames you trained. -
IMPORTANT:
render_pose_trace
,render_test_set
- There are two options for rendering.
- First Option: If you choose
"render_pose_trace": "true"
, then you will get the output video based on your pose trace file. (which should be located inpath_of_MIV_pose_trace_file
)- For now, the pose trace file should be MIV format.
- Second Option: If you choose
"render_test_set": "true"
, then you will get the output video rendered in specific viewing position. (fill intest_set_view
)
2. Run python file
python render.py
3. See Results
- Your output video will be generated in
{result_dir}/train_{train_id}/render_{render_id}/{poses_video or viewN}.mp4
- The rendering log is located in
{result_dir}/train_{train_id}/render_{render_id}/log.txt