DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion

View on arXiv ← Back to list

Authors: Ke Sun, Shen Chen, Taiping Yao, Hong Liu, Xiaoshuai Sun, Shouhong Ding, Rongrong Ji

Published: 2024-10-06 06:22:43+00:00

AI Summary

DiffusionFake enhances deepfake detection model generalization by reversing the deepfake generation process. It injects features from a detection model into a pre-trained Stable Diffusion model to reconstruct source and target images, forcing the detector to learn disentangled representations.

Abstract

The rapid progress of Deepfake technology has made face swapping highly realistic, raising concerns about the malicious use of fabricated facial content. Existing methods often struggle to generalize to unseen domains due to the diverse nature of facial manipulations. In this paper, we revisit the generation process and identify a universal principle: Deepfake images inherently contain information from both source and target identities, while genuine faces maintain a consistent identity. Building upon this insight, we introduce DiffusionFake, a novel plug-and-play framework that reverses the generative process of face forgeries to enhance the generalization of detection models. DiffusionFake achieves this by injecting the features extracted by the detection model into a frozen pre-trained Stable Diffusion model, compelling it to reconstruct the corresponding target and source images. This guided reconstruction process constrains the detection network to capture the source and target related features to facilitate the reconstruction, thereby learning rich and disentangled representations that are more resilient to unseen forgeries. Extensive experiments demonstrate that DiffusionFake significantly improves cross-domain generalization of various detector architectures without introducing additional parameters during inference. Our Codes are available in https://github.com/skJack/DiffusionFake.git.

Key findings

DiffusionFake significantly improves cross-domain generalization of various detector architectures without adding inference parameters. It achieves state-of-the-art results on multiple datasets, particularly showing strong performance against recent diffusion-based face swapping techniques. Ablation studies confirm the importance of all components of the proposed framework.

Approach

DiffusionFake uses a plug-and-play framework. It injects features extracted by a deepfake detection model into a frozen Stable Diffusion model to guide reconstruction of source and target images. This forces the detection model to learn more robust features.

Datasets

FaceForensics++, Celeb-DF, DeepFake Detection (DFD), DFDC Preview (DFDC-P), WildDeepfake, DiffSwap

Model(s)

Stable Diffusion 1.5 (pretrained and frozen), EfficientNet-B4, ViT-B, ResNet-34, EfficientNet-B0, ViT-Small

Author countries

China, China, China, Japan

← Previous