A PyTorch implementation of VSRNet [Video super-resolution with convolutional neural networks] (https://ieeexplore.ieee.org/abstract/document/7444187/)
- Anaconda
- PyTorch
conda install pytorch torchvision -c soumith
conda install pytorch torchvision cuda80 -c soumith # install it if you have installed cuda 8.0
- PyTorchNet
pip install git+https://github.com/pytorch/tnt.git@master
- tqdm
pip install tqdm
- opencv
conda install -c conda-forge opencv
- tensorboard_logger
pip install tensorboard_logger
- h5py
conda install h5py
The train and val datasets are sampled from CDVL dataset. We choose this dataset because we want to extend single frame based SRCNN to multi frame based VSRNet.
Train dataset (uf_X, X=2,3,4) is composed of multiple h5 files:
- Data_CDVL_LR_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from LR frames, 225243x5x(72/X)x(72/X)
- Data_CDVL_HR_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from HR frames, 225243x5x72x72
- Data_CDVL_LR_Bic_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from Matlab bicubic interpolation upscaled frames, 225243x5x72x72
- Data_CDVL_LR_Bic_MC_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from Matlab bicubic interpolation upscaled and optical flow motion compensated frames, 225243x5x72x72
Val dataset (uf_X, X=2,3,4) is composed of multiple h5 files:
- Data_CDVL_LR_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from LR frames, 45159x5x(72/X)x(72/X)
- Data_CDVL_HR_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from HR frames, 45159x5x72x72
- Data_CDVL_LR_Bic_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from Matlab bicubic interpolation upscaled frames, 45159x5x72x72
- Data_CDVL_LR_Bic_MC_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from Matlab bicubic interpolation upscaled and optical flow motion compensated frames, 45159x5x72x72
Test dataset (uf_X, X=2,3,4) is composed of multiple h5 folders:
- LR: LR frames of a scene, (1920/X)x(1080/X)x5xframe_number
- HR: HR frames of a scene, 1920x1080x1xframe_number
- LR_Bic: Matlab Bicubic upscaled LR frames of a scene, 1920x1080x5xframe_number
- LR_Bic_MC: Matlab Bicubic upscaled and Celiu optical flow motion compensated LR frames of a scene, 1920x1080x5xframe_number
- LR_MC: Celiu optical flow motion compensated LR frames of a scene, (1920/X)x(1080/X)x5xframe_number
Download the pre-processed h5 files from Dropbox here or Baidu Yun here with code: apkw , and then setup the path in the codes.
If you want to prepare your own train/val/test h5 files, please check the Matlab scripts in the matlab_pre_processing folder. Notice that you would need to install Celiu optical flow to run the Matlab code.
The test image dataset are sampled from | Set 5 | Bevilacqua et al. BMVC 2012 | Set 14 | Zeyde et al. LNCS 2010 | BSD 100 | Martin et al. ICCV 2001 | Sun-Hays 80 | Sun and Hays ICCP 2012 | Urban 100 | Huang et al. CVPR 2015. Download the preprocessed H5 files from here, and then setup the path in test_image.py file.
SRCNN need to be trained to initialize VSRNet
python train_SRCNN.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 4]
--num_epochs super resolution epochs number [default value is 800]
python train_VSRNet.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 4]
--num_epochs super resolution epochs number [default value is 400]
python test_image_SRCNN.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 3]
--model_name super resolution model name [default value is epochs_SRCNN/model_epoch_800.pth]
The output high resolution images are on results
directory.
python test_video_CDVL_SRCNN.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 3]
--model_name SRCNN model name [default value is epochs_SRCNN/model_epoch_800.pth]
--is_real_time whether to save results into video file or not [default value is False]
--delay_time display delay time [default value is 1]
The output high resolution images are on results
directory.
python test_video_CDVL_VSRNet.py
optional arguments:
--upscale_factor super resolution upscale factor [default value is 3]
--vsrnet_model_name VSRNet model name [default value is epochs_SRCNN/model_epoch_800.pth]
--srcnn_model_name SRCNN model name [default value is epochs_SRCNN/model_epoch_800.pth]
--is_real_time whether to save results into video file or not [default value is False]
--delay_time display delay time [default value is 1]
The output high resolution images are on results
directory.