PyTorch implementation of our ICCV 2019 Oral paper SENSE: A Shared Encoder for Scene-flow Estimation.

Copyright (C) 2019 NVIDIA Corporation. All rights reserved. Licensed under the CC BY-NC-SA 4.0 license (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode).

• Python (tested with Python3.6.10)
• PyTorch (tested with 1.3.0)
• SynchronizedBatchNorm (borrowed from https://github.com/CSAILVision/semantic-segmentation-pytorch)
• tensorboardX
• tqdm
• OpenCV
• scipy
• numpy

It is strongly recommended to use a conda environment to install all the dependencies. Simply run sh scripts/install.sh to install all dependencies and also compile the correlation package.

All experiments were conducted on 8 2080ti GPUs (each with 11GB memory) or 2 M40 GPUs (each with 24GB memory).

In our original implementation, we used a C++ implementation for the cost volume computation for both optical flow and stereo disparity estimations. But the C++ implementatyion strictly requires a PyTorch version of 0.4.0. In this relased version, we switched to use the implemtnation provided at https://github.com/NVIDIA/flownet2-pytorch. We use this implementation for stereo disparity estimation, although it only supports cost volume computation for optical flow (searching for correspondence in a local 2D range). Please consult our paper if you are interested in the running time and GPU memory usuage.

First run sh scripts/download_pretrained_models.sh to download pre-trained models. Run python tools/demo.py then for a quick demo.

Run sh scripits/make_kitti2015_submission.sh to generate results that can be submitted to the online KITTI scene flow estimation benchmark. You should be able to get the following results.

See TRAINING.md for details.

If you find SENSE useful for your research, please consider citing it.

@InProceedings{jiang2019sense,
author = {Jiang, Huaizu and Sun, Deqing and Jampani, Varun and Lv, Zhaoyang and Learned-Miller, Erik and Kautz, Jan},
title = {SENSE: A Shared Encoder Network for Scene-Flow Estimation},
booktitle = {ICCV},
year = {2019}
}