M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection

Authors: Junke Wang, Zuxuan Wu, Wenhao Ouyang, Xintong Han, Jingjing Chen, Ser-Nam Lim, Yu-Gang Jiang

Published: 2021-04-20 05:43:44+00:00

Comment: accepted by ICMR 2022

AI Summary

The paper introduces M2TR, a Multi-modal Multi-scale Transformer for Deepfake detection, designed to capture subtle manipulation artifacts across different spatial levels using RGB and frequency domain information. It employs a carefully designed cross-modality fusion block and is trained with a multi-task objective including segmentation and contrastive loss for improved robustness. Additionally, the research presents SR-DF, a new high-quality Deepfake dataset with 4,000 videos, and demonstrates M2TR's superior performance over state-of-the-art detection methods.

Abstract

The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition, to stimulate Deepfake detection research, we introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods. We conduct extensive experiments to verify the effectiveness of the proposed method, which outperforms state-of-the-art Deepfake detection methods by clear margins.

Key findings

M2TR achieves state-of-the-art Deepfake detection performance across various datasets, including FF++, Celeb-DF, and ForgeryNet, outperforming existing methods by clear margins. The newly introduced SR-DF dataset is validated to be more challenging and diverse than current datasets, providing a strong benchmark for future research. Ablation studies confirm that all components of M2TR (multi-scale transformers, frequency filters, cross-modality fusion, and contrastive loss) significantly contribute to its overall effectiveness and generalization ability.

Approach

The M2TR architecture utilizes a two-stream approach: one stream processes RGB input with multi-scale transformers operating on patches of varying sizes to detect local inconsistencies, while the other extracts forgery features in the frequency domain using learnable filters. Features from both streams are adaptively fused via a cross-modality fusion block. The model is trained with a multi-task loss including binary classification, a segmentation loss for manipulated regions, and a contrastive loss for improved generalization.

Datasets

SR-DF, FaceForensics++ (FF++), Celeb-DF, ForgeryNet, UADFV, DF-TIMIT-LQ, DF-TIMIT-HQ, DFD, WildDeepfake, VoxCeleb (for pre-training models used in SR-DF generation).

Model(s)

M2TR (Multi-modal Multi-scale TRansformer) is the proposed architecture. It utilizes Efficient-b4, pretrained on ImageNet, as its backbone network.

Author countries

China, USA

← Previous