Generalized Deepfakes Detection with Reconstructed-Blended Images and Multi-scale Feature Reconstruction Network

View on arXiv ← Back to list

Authors: Yuyang Sun, Huy H. Nguyen, Chun-Shien Lu, ZhiYong Zhang, Lu Sun, Isao Echizen

Published: 2023-12-13 09:49:15+00:00

AI Summary

This paper proposes a generalized deepfake detection approach using reconstructed blended images (RBIs) for training and a multi-scale feature reconstruction network (MFRN) for detection. RBIs incorporate potential deepfake generator artifacts, while MFRN captures generic boundary artifacts and noise distribution anomalies. This approach improves cross-manipulation and cross-dataset detection performance.

Abstract

The growing diversity of digital face manipulation techniques has led to an urgent need for a universal and robust detection technology to mitigate the risks posed by malicious forgeries. We present a blended-based detection approach that has robust applicability to unseen datasets. It combines a method for generating synthetic training samples, i.e., reconstructed blended images, that incorporate potential deepfake generator artifacts and a detection model, a multi-scale feature reconstruction network, for capturing the generic boundary artifacts and noise distribution anomalies brought about by digital face manipulations. Experiments demonstrated that this approach results in better performance in both cross-manipulation detection and cross-dataset detection on unseen data.

Key findings

The proposed method achieved state-of-the-art or comparable performance in cross-manipulation and cross-dataset detection. The method showed robustness to moderate video compression but was affected by heavy compression. Ablation studies confirmed the importance of each component of the MFRN.

Approach

The approach generates synthetic training samples called reconstructed blended images (RBIs) by blending statistically augmented versions of a genuine image and a reconstructed version of that image, which includes added noise to simulate generator artifacts. A multi-scale feature reconstruction network (MFRN) then learns to detect discrepancies in RGB features, noise patterns, and edge features across multiple scales.

Datasets

FaceForensics++ (FF++) for training; DeepFakeDetection (DFD), Celeb-DF-v2 (CDF-v2), DeepFake Detection Challenge (DFDC), DFDC-P, and FF++ for testing.

Model(s)

Multi-scale Feature Reconstruction Network (MFRN) built upon EfficientNet-B4; SimSwap used for image reconstruction in RBI generation.

Author countries

Japan, Taiwan, China

← Previous