Beyond Face Swapping: A Diffusion-Based Digital Human Benchmark for Multimodal Deepfake Detection

View on arXiv ← Back to list

Authors: Jiaxin Liu, Jia Wang, Saihui Hou, Min Ren, Huijia Wu, Long Ma, Renwang Pei, Zhaofeng He

Published: 2025-05-22 10:46:37+00:00

AI Summary

This paper introduces DigiFakeAV, a large-scale multimodal dataset of diffusion-model-generated deepfakes, highlighting the limitations of existing detection methods on this new benchmark. It also proposes DigiShield, a novel multimodal deepfake detection framework that achieves state-of-the-art performance on DigiFakeAV and generalizes well to other datasets.

Abstract

In recent years, the explosive advancement of deepfake technology has posed a critical and escalating threat to public security: diffusion-based digital human generation. Unlike traditional face manipulation methods, such models can generate highly realistic videos with consistency via multimodal control signals. Their flexibility and covertness pose severe challenges to existing detection strategies. To bridge this gap, we introduce DigiFakeAV, the new large-scale multimodal digital human forgery dataset based on diffusion models. Leveraging five of the latest digital human generation methods and a voice cloning method, we systematically construct a dataset comprising 60,000 videos (8.4 million frames), covering multiple nationalities, skin tones, genders, and real-world scenarios, significantly enhancing data diversity and realism. User studies demonstrate that the misrecognition rate by participants for DigiFakeAV reaches as high as 68%. Moreover, the substantial performance degradation of existing detection models on our dataset further highlights its challenges. To address this problem, we propose DigiShield, an effective detection baseline based on spatiotemporal and cross-modal fusion. By jointly modeling the 3D spatiotemporal features of videos and the semantic-acoustic features of audio, DigiShield achieves state-of-the-art (SOTA) performance on the DigiFakeAV and shows strong generalization on other datasets.

Key findings

Existing deepfake detection models show significant performance degradation on the new DigiFakeAV dataset. DigiShield achieves state-of-the-art performance on DigiFakeAV, demonstrating the effectiveness of multimodal spatiotemporal fusion. User studies show a high misrecognition rate (68%) on DigiFakeAV, highlighting the challenge posed by these advanced deepfakes.

Approach

DigiShield, the proposed detection framework, uses a spatiotemporal two-stream pipeline to extract features from video and audio. It then employs cross-modal and self-attention mechanisms to fuse these features, capturing inconsistencies between audio and video indicative of deepfakes. Finally, a classification layer determines the authenticity of the input.

Datasets

DigiFakeAV (created by the authors), HDTF, CelebV-HQ, FF++, Celeb-DF, DFDC, DF-TIMIT, FakeAVCeleb, DeepSpeak

Model(s)

3D-ResNet-50 (backbone for DigiShield), various pre-existing deepfake detection models for comparison (Meso4, MesoInception4, Xception-c23, Capsule, HeadPose, F3-Net, Cross Efficient ViT, SSVF, SFIConv)

Author countries

China

← Previous