Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing

View on arXiv ← Back to list

Authors: Xinghe Fu, Zhiyuan Yan, Taiping Yao, Shen Chen, Xi Li

Published: 2025-01-08 09:30:45+00:00

AI Summary

This paper addresses the generalization problem in deepfake detection by identifying and mitigating position and content biases in existing detectors. The authors propose a novel method, UDD, which uses token-level shuffling and mixing in the latent space of transformers to learn unbiased forgery representations.

Abstract

The generalization problem is broadly recognized as a critical challenge in detecting deepfakes. Most previous work believes that the generalization gap is caused by the differences among various forgery methods. However, our investigation reveals that the generalization issue can still occur when forgery-irrelevant factors shift. In this work, we identify two biases that detectors may also be prone to overfitting: position bias and content bias, as depicted in Fig. 1. For the position bias, we observe that detectors are prone to lazily depending on the specific positions within an image (e.g., central regions even no forgery). As for content bias, we argue that detectors may potentially and mistakenly utilize forgery-unrelated information for detection (e.g., background, and hair). To intervene these biases, we propose two branches for shuffling and mixing with tokens in the latent space of transformers. For the shuffling branch, we rearrange the tokens and corresponding position embedding for each image while maintaining the local correlation. For the mixing branch, we randomly select and mix the tokens in the latent space between two images with the same label within the mini-batch to recombine the content information. During the learning process, we align the outputs of detectors from different branches in both feature space and logit space. Contrastive losses for features and divergence losses for logits are applied to obtain unbiased feature representation and classifiers. We demonstrate and verify the effectiveness of our method through extensive experiments on widely used evaluation datasets.

Key findings

UDD achieves state-of-the-art results in cross-dataset evaluation, significantly outperforming existing methods on multiple datasets. The method also demonstrates improved robustness against various image perturbations. Ablation studies confirm the effectiveness of both the shuffling and mixing branches in reducing bias and improving generalization.

Approach

The proposed UDD method uses two branches: a shuffling branch that rearranges tokens and position embeddings, and a mixing branch that mixes tokens between images of the same label. Contrastive and divergence losses align outputs from these branches to create unbiased feature representations and classifiers.

Datasets

FaceForensics++, Celeb-DF, DFDC, DFDC-Preview, DFD, WildDeepfake, FaceShifter, FFIW

Model(s)

ViT-B (Vision Transformer - Base), CLIP (Contrastive Language-Image Pre-training)

Author countries

China

← Previous