Yiqing Liang, Eliot Laidlaw, Alexander Meyerowitz, Srinath Sridhar, James Tompkin

Official Implementation for "Semantic Attention Flow Fields for Monocular Dynamic Scene Decomposition".

[Paper] [Project] [Data]

Tested with System Spec:

python==3.7.4
cmake==3.20.0
gcc==10.2
cuda==11.3.1
cudnn==8.2.0 

Steps:

python3 -m venv <env_name>
source <env_name>/bin/activate

pip install --upgrade pip

pip install -r [path/to/repo]/requirements.txt

# install eigen using apt-get if can; otherwise compile from source:
cd <folder to store tmp file>
wget https://gitlab.com/libeigen/eigen/-/archive/master/eigen-master.tar.gz
tar -xzf eigen-master.tar.gz
cd eigen-master
rm -rf build
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=<env_name>
make -j8
make install

#install pydensecrf
pip3 install --force-reinstall cython==0.29.36
pip install --no-build-isolation git+https://github.com/lucasb-eyer/pydensecrf.git

We tested our code with 1 single RTX3090 / A6000.

NVIDIA dataset

gt_masks.zip: segmentation annotation for NVIDIA dataset

ours_1018_processed_crf.zip: final results on NVIDIA dataset

nvidia_data_full.zip: processed NVIDIA dataset for training and testing

checkpoints.zip: trained models for NIVIDIA dataset

For usage, Data and Checkpoints should be put to corresponding location according to config files.

cd [path/to/repo]/Neural-Scene-Flow-Fields/

Data folder should be renamed as datadir.

Checkpoint should be put under basedir/expname.

cd [path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp

python run_nerf.py --config [path/to/config/file]

For now, each scene([path/to/config/file])'s corresponding results is stored to:

├── [path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/
│   ├── 000000.tar
│   ├── 010000.tar
│   ├── ...
│   ├── 360000.tar
│   ├── args.txt
│   ├── config.txt

python run_nerf.py --config [path/to/config/file] --render_2D

This would create a new folder under scene folder:

├── [path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/
|   ├──render_2D-010_path_[last_ckpt_id+1]
|       ├──0_blend.png
|       ├──0_depth.png
|       ├──0_dino.pt
|       ├──0_rgb.png
|       ├──0_sal.png
|       ├──...

• 0-23: training views

• 24-47: fixed camera 0, moving times

• 48-59: moving camera, fixed time = 0

python run_nerf.py --config [path/to/config/file] --cluster_2D

python run_nerf.py --config [path/to/config/file] --cluster_2D --render_mode

This would create a new folder under scene folder:

├── [path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/
|   ├──cluster_2D-010_path_[last_ckpt_id+1]
|       ├──final
|           ├──0.png
|           ├──1.png
|           ├──...
|       ├──no_merge_no_salient
|           ├──0.png
|           ├──1.png
|           ├──...
|       ├──no_salient
|           ├──0.png
|           ├──1.png
|           ├──...
|       ├──0_bld_full_beforemerge.png
|       ├──0_bld_full.png
|       ├──0_bld.pt
|       ├──0_clu_full_beforemerge.png
|       ├──0_clu_full.png
|       ├──0_clu.png
|       ├──...

python postprocess_oracle.py --raw_folder [path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/cluster_2D-010_path_[last_ckpt_id+1]/no_salient --gt_folder [path/to/this/scene/mask] [--flip_fg]

If use Black to denote background, need --flip_fg flag.

This would create a new folder oracle under no_salient folder.

Note: [path/to/your/final/result] could be your default clustering result folder [path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/cluster_2D-010_path_[last_ckpt_id+1]/final or oracle processed folder [path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/cluster_2D-010_path_[last_ckpt_id+1]/no_salient/oracle.

python postprocess_per_scene.py --root_dir [path/to/your/final/result]

python postprocess_crf_per_scene.py --root_dir [path/to/your/final/result]_processed --render_dir [path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/render_2D-010_path_[last_ckpt_id+1]

First organize all scene's final clustering result([path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/cluster_2D-010_path_[last_ckpt_id+1]/final) and final rgb rendering result([path/to/repo]/Neural-Scene-Flow-Fields/nsff_exp/logs/[your/exp/name]_F[start_frame]_[end_frame]/render_2D-010_path_[last_ckpt_id+1]/*_rgb.png) under the same folder:

├── [path/to/results]
|   ├──Balloon1-2
|       ├──0.png # cluster result
|       ├──0_rgb.png # rgb result
|       ├──...
|   ├──Balloon2-2
|   ├──DynamicFace-2
|   ├──Jumping
|   ├──playground
|   ├──Skating-2
|   ├──Truck-2
|   ├──Umbrella-2

for now only supports NVIDIA scene organization!

python postprocess.py --root_dir [path/to/results]

python postprocess_crf.py --root_dir [path/to/results]_processed

Now the results would be stored in folder [path/to/results]_processed_crf.

for now only supports NVIDIA scene organization!

cd [path/to/repo]/benchmarks

python evaluate_fg_ours.py --vis_folder [path/to/results]_processed_crf --gt_folder [path/to/gt_masks]

python evaluate_ours.py --vis_folder [path/to/results]_processed_crf --gt_folder [path/to/gt_masks]

Collect rgb image sequence and corresponding dynamic masks, organize like:

├── [path/to/data]
|   ├──images
|       ├──00000.png 
|       ├──00001.png
|       ├──...
|   ├──colmap_masks
|       ├──00000.png.png
|       ├──00001.png.png
|       ├──...

Note: Naming has to follow %05d format!

colmap_masks is where we store foreground masks. Dynamic Foreground -> black. Static Background -> white.

Make sure you have colmap installed on your machine.

For example, on Mac, run sudo zsh colmap.sh [path/to/data]

After running, the same data folder looks like:

├── [path/to/data]
|   ├──images
|       ├──...
|   ├──colmap_masks
|       ├──...
|   ├──dense
|       ├──...
|   ├──sparse
|       ├──...
|   ├──database.db
|   ├──database.db-shm
|   ├──database.db-wal

[path/to/data]/dense is the final data we want.

cd [path/to/repo]/Neural-Scene-Flow-Fields/nsff_scripts

python save_poses_nerf.py --data_path "[path/to/data]/dense/"

Download single view depth prediction model model.pt from link, and put it under the folder nsff_scripts.

python run_midas.py --data_path "[path/to/data]/dense/" [--resize_height ???] # use resize_height if data is over big for SAFF

Download RAFT model raft-things.pth from link, and put it under the folder nsff_scripts/models.

python run_flows_video.py --model models/raft-things.pth --data_path [path/to/data]/dense/

After running, the same data folder looks like:

├── [path/to/data]
|   ├──images
|       ├──...
|   ├──colmap_masks
|       ├──...
|   ├──dense
|       ├──disp
|           ├──...
|       ├──flow_i1
|           ├──...
|       ├──images
|           ├──...
|       ├──images_306x400
|           ├──...
|       ├──motion_masks
|           ├──...
|       ├──sparse
|           ├──...
|       ├──stereo
|           ├──...
|       ├──poses_bounds.npy
|       ├──run-colmap-geometric.sh
|       ├──run-colmap-photometric.sh
|       ├──scene.json
|   ├──sparse
|       ├──...
|   ├──database.db
|   ├──database.db-shm
|   ├──database.db-wal

copy config file template, and change the value of field:

• expname: to be your expname
• datadir: to your [path/to/data]/dense
• final_height: this must be same as --resize_height argument in run_midas.py
• start_frame, end_frame: which images would participate in training according to image id.

Note: if end_frame - start_frame == 48, would filter out half of the images for testing. (for DyCheck's sake)

• https://github.com/gaochen315/DynamicNeRF
• https://github.com/zhengqili/Neural-Scene-Flow-Fields/tree/main