[Paper]
Ying Xu, Kiran Raja, Luisa Verdoliva, Marius Pedersen
We propose a new approach, Multi-Channel Xception Attention Pairwise Interaction (MCX-API), that exploits the power of pairwise learning and complementary information from different color space representations in a fine-grained manner. We first validate our idea on a publicly available dataset in a intra-class setting (closed set) with four different Deepfake schemes. Further, we report all the results using balanced-open-set-classification (BOSC) accuracy in an inter-class setting (open-set) using three public datasets. Our experiments indicate that our proposed method can generalize better than the state-of-the-art Deepfakes detectors. We obtain 98.48% BOSC accuracy on the FF++ dataset and 90.87% BOSC accuracy on the CelebDF dataset suggesting a promising direction for generalization of DeepFake detection. We further utilize t-SNE and attention maps to interpret and visualize the decision-making process of our proposed network.
If you want to test, please refer to test.slurm for examples.
For this model, I extracted 1 frame per 10 frames. Code could be referred to this link. Then you can use create_list_FF++.py to create train.txt or test.txt for training and test.
The train.txt or test.txt include 'image_path label' every line. Here is an example:
FaceForensics++/original_sequences/youtube/c23/face_images/870/frame121.png 0
FaceForensics++/manipulated_sequences/Deepfakes/c23/face_images/979_875/frame1.png 1
...
Here is the link for MCX-API model for RGB.
Please kindly cite the following paper, if you find this code helpful in your work.
@inproceedings{xu2023learning,
title={Learning Pairwise Interaction for Generalizable DeepFake Detection},
author={Xu, Ying and Raja, Kiran and Verdoliva, Luisa and Pedersen, Marius},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={672--682},
year={2023}
}
@inproceedings{zhuang2020learning,
title={Learning Attentive Pairwise Interaction for Fine-Grained Classification.},
author={Zhuang, Peiqin and Wang, Yali and Qiao, Yu},
booktitle={AAAI},
pages={13130--13137},
year={2020}
}
Please feel free to contact [email protected], if you have any questions.