Rethinking Video Frame Interpolation from Shutter Mode Induced Degradation

Paper | Supp | Video

Xiang Ji¹, Zhixiang Wang^1,2, Zhihang Zhong^1,2, Yinqiang Zheng¹

¹The University of Tokyo  ²National Institute of Informatics  

TL;DR

In this paper, we present the first real-world dataset for learning and benchmarking degraded video frame interpolation, named RD-VFI, and further explore the performance differences of three types of degradations, including GS blur, RS distortion, and an in-between effect caused by the rolling shutter with global reset (RSGR), thanks to our novel quad-axis imaging system. Moreover, we propose a unified Progressive Mutual Boosting Network (PMBNet) model to interpolate middle frames at arbitrary time for all shutter modes. Its disentanglement strategy and dual-stream correction enable us to adaptively deal with different degradations for VFI.

Dependencies

1. Python and Pytorch

• Pyhotn=3.8 (Anaconda recommended)
• Pytorch=1.11.0
• CUDA=11.3/11.4

conda create -n pmbnet python=3.8
conda activate pmbnet
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch

####  please install detron2 using command as bellow
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

2. Other packages

pip install -r requirements.txt

Data and Pretrained Model

• Download datasets RD_VFI and synthetic data GOPRO-VFI_copy based on GOPRO.
• Unzip them under a specified directory by yourself.
• Please download checkpoints from this link and put them under root directory of this project.

Test

To test PMBNet, please run the command below:

bash ./test.sh       ### Please specify your data directory, data mode and output path in the script

Train

To train PMBNet, please run the command below:

bash ./train.sh       ### Please refer to the script for more info.

Citation

If you find our work useful, please kindly cite as:

@InProceedings{Ji_2023_ICCV,
    author    = {Ji, Xiang and Wang, Zhixiang and Zhong, Zhihang and Zheng, Yinqiang},
    title     = {Rethinking Video Frame Interpolation from Shutter Mode Induced Degradation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {12259-12268}
}

Acknowledgement

This project is based on works below:

• RIFE
• MIMO-UNet
• DEMFI and the public dataset GOPRO