Xiang Ji1, Zhixiang Wang1,2, Zhihang Zhong1,2, Yinqiang Zheng1
1The University of Tokyo 2National Institute of Informatics
In this paper, we present the first real-world dataset for learning and benchmarking degraded video frame interpolation, named RD-VFI, and further explore the performance differences of three types of degradations, including GS blur, RS distortion, and an in-between effect caused by the rolling shutter with global reset (RSGR), thanks to our novel quad-axis imaging system. Moreover, we propose a unified Progressive Mutual Boosting Network (PMBNet) model to interpolate middle frames at arbitrary time for all shutter modes. Its disentanglement strategy and dual-stream correction enable us to adaptively deal with different degradations for VFI.
- Python and Pytorch
- Pyhotn=3.8 (Anaconda recommended)
- Pytorch=1.11.0
- CUDA=11.3/11.4
conda create -n pmbnet python=3.8
conda activate pmbnet
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch
#### please install detron2 using command as bellow
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
- Other packages
pip install -r requirements.txt
- Download datasets RD_VFI and synthetic data GOPRO-VFI_copy based on GOPRO.
- Unzip them under a specified directory by yourself.
- Please download checkpoints from this link and put them under root directory of this project.
To test PMBNet, please run the command below:
bash ./test.sh ### Please specify your data directory, data mode and output path in the script
To train PMBNet, please run the command below:
bash ./train.sh ### Please refer to the script for more info.
If you find our work useful, please kindly cite as:
@InProceedings{Ji_2023_ICCV,
author = {Ji, Xiang and Wang, Zhixiang and Zhong, Zhihang and Zheng, Yinqiang},
title = {Rethinking Video Frame Interpolation from Shutter Mode Induced Degradation},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {12259-12268}
}
This project is based on works below: