Learning Pairwise Interaction for Generalizable DeepFake Detection

View on arXiv ← Back to list

Authors: Ying Xu, Kiran Raja, Luisa Verdoliva, Marius Pedersen

Published: 2023-02-26 10:39:08+00:00

AI Summary

The paper proposes MCX-API, a deepfake detection method that leverages pairwise learning and multi-channel color space representations to improve generalizability. Experiments show MCX-API outperforms state-of-the-art methods on both seen and unseen deepfake datasets, achieving high BOSC accuracy.

Abstract

A fast-paced development of DeepFake generation techniques challenge the detection schemes designed for known type DeepFakes. A reliable Deepfake detection approach must be agnostic to generation types, which can present diverse quality and appearance. Limited generalizability across different generation schemes will restrict the wide-scale deployment of detectors if they fail to handle unseen attacks in an open set scenario. We propose a new approach, Multi-Channel Xception Attention Pairwise Interaction (MCX-API), that exploits the power of pairwise learning and complementary information from different color space representations in a fine-grained manner. We first validate our idea on a publicly available dataset in a intra-class setting (closed set) with four different Deepfake schemes. Further, we report all the results using balanced-open-set-classification (BOSC) accuracy in an inter-class setting (open-set) using three public datasets. Our experiments indicate that our proposed method can generalize better than the state-of-the-art Deepfakes detectors. We obtain 98.48% BOSC accuracy on the FF++ dataset and 90.87% BOSC accuracy on the CelebDF dataset suggesting a promising direction for generalization of DeepFake detection. We further utilize t-SNE and attention maps to interpret and visualize the decision-making process of our proposed network. https://github.com/xuyingzhongguo/MCX-API

Key findings

MCX-API achieved 98.48% BOSC accuracy on the FF++ dataset and 90.87% on Celeb-DF, significantly outperforming state-of-the-art methods. The approach demonstrates improved generalizability to unseen deepfake types, showcasing the effectiveness of pairwise learning and multi-channel feature extraction.

Approach

MCX-API uses a multi-channel Xception network to extract features from image pairs represented in different color spaces. Pairwise interaction and attention mechanisms are employed to learn discriminative features, improving detection accuracy and generalizability to unseen deepfake generation methods.

Datasets

FaceForensics++, Celeb-DF, KoDF, FakeAV

Model(s)

Multi-Channel Xception Attention Pairwise Interaction (MCX-API) network, MTCNN (for face detection and alignment)

Author countries

Norway, Italy

← Previous