Skip to content

Code for Deep Surface Normal Estimation with Hierarchical RGB-D Fusion (CVPR2019)

Notifications You must be signed in to change notification settings

jzengust/RGBD2Normal

Repository files navigation

CVPR2019 Deep Surface Normal Estimation with Hierarchical RGB-D Fusion

Paper

Deep Surface Normal Estimation with Hierarchical RGB-D Fusion (CVPR2019) [Project page]

Architecture

File Organization

|-- models: model structures
|-- pre_trained: pretrain models
|-- loader: dataloaders
|-- datalist: train/test list for matterport and scannet
              testsmall: used for fast checking
|-- checkpoint/
    FCONV_MS: RGBD model for 
        |-- matterport (fconv_ms_matterport_l1_2_hybrid_best.pkl) 
        |-- scannet (fconv_ms_scannet_l1_1_hybrid_best.pkl)
    RGB_resume: RGB model for 
        |-- matterport (vgg_16_in_matterport_l1_2_in_RGB_best.pkl) 
        |-- scannet (vgg_16_in_scannet_l1_2_resume_RGB_best.pkl)
|-- sample_bash: bash sample for training and testing
|-- demo_rgbd_mp.sh/demo_rgbd_sc.sh , demo script
|-- sample_pic: sample pictures for demo
|-- train_RGB.py, training for RGB model 
|-- test_RGB.py, testing for RGB model
|-- train_RGBD_ms.py, training for RGBD model
|-- test_RGBD_ms.py, testing for RGBD model
|-- test_RGBD_ms_sc.py, testing for RGBD model for object details
|-- config.json, config file for dataset filepath.
|-- utils.py, misc

Note: due to upload size limit, pretrain model and checkpoint are not uploaded. Please refer to Model section for model download.

Requirement

  • python 2.7
  • torch 0.4.1
  • torchvision 0.2.1
  • scikit-learn 0.20.3
  • numpy
  • matplotlib
  • tqdm
  • tensorboardX

Dataset

Two datasets are used: Matterport and Scannet. Please refer to [this website] for data download.
After data download, please config the data path in config.json.

Model

Download the model and pretrain file from [Google Drive]

Demo

  1. Please set up environment and download the model.
  2. run demo_rgbd_mp.sh. Result will be saved in ./result/demo_rgbd_mp/
  3. run demo_rgbd_sc.sh. Result will be saved in ./result/demo_rgbd_sc/

Usage

Please set the data path in config.json

Sample bash files can be found in ./sample_bash for training and testing. You may use them directly or follow the below instruction to make your own training/testing scripts.

To train/test on matterport,

  1. training:

    • a. train RGB model with l1 loss (train_RGB.py)
    • b. train fusion network with map_conv (no hybrid, just l1 at final scale) use RGB model for RGB encoder several epoch will be okay (5~10)
    • c. use the model above as pretrained model, apply hybrid (5~10)
  2. testing: run test_RGBD.py
    func eval_normal_pixel for evaluation metric computation
    Metric result saved in ./model_path/model_name.csv, as the below sequence:

    Mean error|Median error|Percentage of error less than 11.25°|22.5°|30°

    The average result is stored in the last row. For metric detail meaning, please refer to paper

To train/test on scannet

  1. training:

    • a. train RGB model with l1 loss (train_RGB.py)
    • b. train fusion network with map_conv (train_RGBD_ms.py, no hybrid, just l1 at final scale) use RGB model for RGB encoder several epoch will be okay (5~10)
    • c. use the model above as pretrained model, apply hybrid(train_RGBD_ms.py,5~10 epoch)
  2. testing: run test_RGBD_ms.py
    func eval_normal_pixel for evaluation metric computation
    Average metric result saved in ./model_path/model_name.csv

  3. Additionally, run test_RGBD_ms_sc.py
    For object detail normal computation. Pixels belonging to bed, sofa, chair are computed separately

Citation

    @inproceedings{zeng2019deep,
      title={Deep Surface Normal Estimation with Hierarchical RGB-D Fusion},
      author={Zeng, Jin and Tong, Yanfeng and Huang, Yunmu and Yan, Qiong and Sun, Wenxiu and Chen, Jing and Wang, Yongtian},
      booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
      year={2019}
    } 

Contact

Jin Zeng, [email protected]
Yunmu Huang, [email protected]

About

Code for Deep Surface Normal Estimation with Hierarchical RGB-D Fusion (CVPR2019)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published