This repository (https://github.com/twhui/LiteFlowNet) is the offical release of LiteFlowNet for my paper LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation in CVPR18 (Spotlight). The up-to-date version of the paper is available on arXiv.

LiteFlowNet is a lightweight, fast, and accurate opitcal flow CNN. We develop several specialized modules including pyramidal features, cascaded flow inference (cost volume + sub-pixel refinement), feature warping (f-warp) layer, and flow regularization by feature-driven local convolution (f-lcon) layer. LiteFlowNet outperforms PWC-Net (CVPR18) on KITTI and has a smaller model size.

For more details about LiteFlowNet, you may visit my project page.

NEW! Our extended work (LiteFlowNet2) is now available at https://github.com/twhui/LiteFlowNet2.

All code and other materials (including but not limited to the paper, figures, and tables) are provided for research purposes only and without any warranty. Any commercial use requires our consent. When using any parts of the code package or the paper (LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation) in your work, please cite the following paper:

@InProceedings{hui18liteflownet,    
 author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},    
 title = {LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation},    
 booktitle  = {Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},    
 year = {2018},    
 url = {http:https://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 
}

1. FlyingChairs dataset (31GB) and train-validation split.
2. RGB image pairs (clean pass) (37GB) and flow fields (311GB) for Things3D dataset.
3. Sintel dataset (clean + final passes) (5.3GB).
4. KITTI12 dataset (2GB) and KITTI15 dataset (2GB) (Simple registration is required).

A PyTorch-based reimplementation of LiteFlowNet is now available at https://github.com/sniklaus/pytorch-liteflownet.

The code package comes as the modified Caffe from DispFlowNet and FlowNet2 with our new layers, scripts, and trained models.

Installation was tested under Ubuntu 14.04.5/16.04.2 with CUDA 8.0, cuDNN 5.1 and openCV 2.4.8/3.1.0.

Edit Makefile.config (and Makefile) if necessary in order to fit your machine's settings.

For openCV 3+, you may need to change opencv2/gpu/gpu.hpp to opencv2/cudaarithm.hpp in /src/caffe/layers/resample_layer.cu.

If your machine installed a newer version of cuDNN, you do not need to downgrade it. You can do the following trick:

1. Download cudnn-8.0-linux-x64-v5.1.tgz and untar it to a temp folder, say cuda-8-cudnn-5.1

2. Rename cudnn.h to cudnn-5.1.h in the folder /cuda-8-cudnn-5.1/include

3. $ sudo cp cuda-8-cudnn-5.1/include/cudnn-5.1.h /usr/local/cuda/include/	
   $ sudo cp cuda-8-cudnn-5.1/lib64/lib* /usr/local/cuda/lib64/

4. Replace #include <cudnn.h> to #include <cudnn-5.1.h> in /include/caffe/util/cudnn.hpp.

$ cd LiteFlowNet
$ make -j 8 tools pycaffe

The source files include /src/caffe/layers/warp_layer.cpp, /src/caffe/layers/warp_layer.cu, and /include/caffe/layers/warp_layer.hpp.

The grid pattern that is used by f-warp layer is generated by a grid layer. The source files include /src/caffe/layers/grid_layer.cpp and /include/caffe/layers/grid_layer.hpp.

It is implemented using off-the-shelf components. More details can be found in /models/testing/depoly.prototxt or /models/training_template/train.prototxt.template by locating the code segment NetE-R.

Two custom layers (ExpMax and NegSquare) are optimized in speed for forward-pass. Generalized Charbonnier loss is implemented in l1loss_layer. The power factor (alpha) can be adjusted in l1_loss_param { power: alpha l2_per_location: true }.

1. Prepare the training set. In /data/make-lmdbs-train.sh, change YOUR_TRAINING_SET and YOUR_TESTING_SET to your favourite dataset.

$ cd LiteFlowNet/data
$ ./make-lmdbs-train.sh

2. Copy files from /models/training_template to a new model folder (e.g. NEW). Edit all the files and make sure the settings are correct for your application. Model for the complete network is provided. LiteFlowNet uses stage-wise training to boost the performance. Please refer to my paper for more details.

$ mkdir LiteFlowNet/models/NEW
$ cd LiteFlowNet/models/NEW
$ cp ../training_template/solver.prototxt.template solver.prototxt	
$ cp ../training_template/train.prototxt.template train.prototxt
$ cp ../training_template/train.py.template train.py

3. Create a soft link in your new model folder

$ ln -s ../../build/tools bin

4. Run the training script

$ ./train.py -gpu 0 2>&1 | tee ./log.txt

The trained models (liteflownet, liteflownet-ft-sintel, liteflownet-ft-kitti) are available in the folder /models/trained. Untar the files to the same folder before you use it.

liteflownet: Trained on Chairs and then fine-tuned on Things3D.

liteflownet-ft-sintel: Model used for Sintel benchmark.

liteflownet-ft-kitti: Model used for KITTI benchmark.

1. Open the testing folder

$ cd LiteFlowNet/models/testing

2. Create a soft link in the folder /testing

$ ln -s ../../build/tools bin

3. Replace MODE in ./test_MODE.py to batch if all the images has the same resolution (e.g. Sintel dataset), otherwise replace it to iter (e.g. KITTI dataset).

4. Replace MODEL in line 10 (cnn_model = 'MODEL') of test_MODE.py to one of the trained models (e.g. liteflownet-ft-sintel).

5. Run the testing script. Flow fields (MODEL-0000000.flo, MODEL-0000001.flo, ... etc) are stored in the folder /testing/results having the same order as the image pair sequence.

$ test_MODE.py img1_pathList.txt img2_pathList.txt results

Average end-point error can be computed using the provided script /models/testing/util/endPointErr.m