Video restoration Processing Pipeline

As the name suggests, this is a video restoration pipeline pulling from various cutting-edge technologies and merging them to create one processing pipeline, for videos, to rule them all. The pipeline borrows from multiple AI techniques from different contributers, these techniques are mentioned in our releases page. If you like our project please give us a star and also don't forget to like the other projects used by the video restoration pipeline 🤠

NOTE only one video at a time

Setting up the environment

# Make sure you have git installed
git clone https://github.com/cliffordkleinsr/DE-SRFREN.git
cd DE-SRFREN
# Make sure you have python and pytorch installed -.-"
# Install basicsr 
pip install basicsr 
# Install facexlib 
# We use face detection and face restoration helper in the facexlib package
pip install facexlib #parsing path net and resnet faces
pip install realesrgan  
pip install gfpgan
pip install -r requirements.txt

As a side note, make sure you have Pytorch compiled with Cuda binaries installed otherwise inference speed will be greatly impacted

• Basic argument structure:

-i or --input, your input video directory
-o or --video_output, your video output
-n, model name
--ffmpeg_bin, path to ffmpeg.exe
--ffprobe_bin, path to fprobe.exe
-h or --help, for help with arguments

• For quick inference on Windows

cd v0.0.2
python inference.py -i inputs/your_video.mp4 --ffmpeg_bin ffmpeg/bin/ffmpeg.exe --ffprobe_bin ffmpeg/bin/ffprobe.exe --face_enhance --suffix outx2 

Note: face_enhancer only works with videos of real people, If you are working with anime/animation (cartoon) characters, use:

python inference.py -i inputs/your_anime_video.mp4 --ffmpeg_bin ffmpeg/bin/ffmpeg.exe --ffprobe_bin ffmpeg/bin/ffprobe.exe -n realesr-animevideov3 --suffix outx2

• For quick inference on Colab/Linux environment is similar to Windows but avoid using the --ffmpeg_bin and --ffprobe_bin when the binaries are already installed
• The Vector Quantized Code book is deprecated and thus can only be used with v0.0.1:

Original	Processed

Original	Processed

1. Take a video frame and turn into images
2. Super resolve the image
3. Restore the Faces in each frame step
4. Merge Frames H.264 codec MP4
5. Speed up inference Uses NVENC PIPE: Inference now at: 10fps using RTX 4090 | 7fps using RTX 4080 | 3 fps using RTX 3060. Tested using these three GPU's!

1. Frame Generation 24-60 FPS
2. More support for different video formats
3. Color BLack and White Images
4. Lossless Decoding and encoding
5. Sound restoration

@InProceedings{clifford2023desrfren,
   author = {Clifford Njoroge},
   title = {DE-SRFREN: Video restoration Processing Pipeline},
   date = {2023}
 }

Real-ESRGAN

@InProceedings{wang2021realesrgan,
   author    = {Xintao Wang and Liangbin Xie and Chao Dong and Ying Shan},
   title     = {Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data},
   booktitle = {International Conference on Computer Vision Workshops (ICCVW)},
   date      = {2021}
}

VQFR

@inproceedings{gu2022vqfr,
  title={VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder},
  author={Gu, Yuchao and Wang, Xintao and Xie, Liangbin and Dong, Chao and Li, Gen and Shan, Ying and Cheng, Ming-Ming},
  year={2022},
  booktitle={ECCV}
}

GFPGAN

@InProceedings{wang2021gfpgan,
    author = {Xintao Wang and Yu Li and Honglun Zhang and Ying Shan},
    title = {Towards Real-World Blind Face Restoration with Generative Facial Prior},
    booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2021}
}

IMAGEIO