DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder

View on arXiv ← Back to list

Authors: Peipeng Yu, Hui Gao, Jianwei Fei, Zhitao Huang, Zhihua Xia, Chip-Hong Chang

Published: 2024-12-10 07:42:02+00:00

AI Summary

DFREC is a novel DeepFake Identity Recovery scheme that reconstructs both source and target faces from a deepfake image, improving interpretability and enabling identity tracing for forensic investigation. It uses an Identity Segmentation Module, a Source Identity Reconstruction Module, and a Target Identity Reconstruction Module, leveraging a Masked Autoencoder for high-fidelity recovery.

Abstract

Recent advances in deepfake forensics have primarily focused on improving the classification accuracy and generalization performance. Despite enormous progress in detection accuracy across a wide variety of forgery algorithms, existing algorithms lack intuitive interpretability and identity traceability to help with forensic investigation. In this paper, we introduce a novel DeepFake Identity Recovery scheme (DFREC) to fill this gap. DFREC aims to recover the pair of source and target faces from a deepfake image to facilitate deepfake identity tracing and reduce the risk of deepfake attack. It comprises three key components: an Identity Segmentation Module (ISM), a Source Identity Reconstruction Module (SIRM), and a Target Identity Reconstruction Module (TIRM). The ISM segments the input face into distinct source and target face information, and the SIRM reconstructs the source face and extracts latent target identity features with the segmented source information. The background context and latent target identity features are synergetically fused by a Masked Autoencoder in the TIRM to reconstruct the target face. We evaluate DFREC on six different high-fidelity face-swapping attacks on FaceForensics++, CelebaMegaFS and FFHQ-E4S datasets, which demonstrate its superior recovery performance over state-of-the-art deepfake recovery algorithms. In addition, DFREC is the only scheme that can recover both pristine source and target faces directly from the forgery image with high fadelity.

Key findings

DFREC outperforms state-of-the-art deepfake recovery algorithms in recovering both source and target faces with high fidelity across multiple datasets and forgery methods. It achieves superior performance in terms of Frechet Inception Distance (FID) and Identity Similarity (IDSim). The ablation study confirms the contribution of each module and loss function.

Approach

DFREC employs three modules: an Identity Segmentation Module to separate source and target face information; a Source Identity Reconstruction Module to recover the source face and extract latent target features; and a Target Identity Reconstruction Module using a Masked Autoencoder to reconstruct the target face by fusing background context and latent target features.

Datasets

FaceForensics++, CelebaMegaFS, FFHQ-E4S

Model(s)

DeepLabv3, EfficientNet-B0, U-Net-like model, Vision Transformer (ViT), Masked Autoencoder (MAE), FaceNet, VGG19

Author countries

China, China, China, China, China, Singapore

← Previous