Towards Generalizable Deepfake Detection with Locality-aware AutoEncoder

Authors: Mengnan Du, Shiva Pentyala, Yuening Li, Xia Hu

Published: 2019-09-13 02:01:32+00:00

Comment: Accepted by CIKM 2020

AI Summary

This paper introduces the Locality-Aware AutoEncoder (LAE) to enhance the generalization capability of deepfake detection on previously unseen manipulations. LAE enforces models to learn intrinsic representations from forgery regions by regularizing local interpretation with pixel-wise masks. An active learning framework is proposed to significantly reduce the annotation efforts for these masks, requiring less than 3% of the training data.

Abstract

With advancements of deep learning techniques, it is now possible to generate super-realistic images and videos, i.e., deepfakes. These deepfakes could reach mass audience and result in adverse impacts on our society. Although lots of efforts have been devoted to detect deepfakes, their performance drops significantly on previously unseen but related manipulations and the detection generalization capability remains a problem. Motivated by the fine-grained nature and spatial locality characteristics of deepfakes, we propose Locality-Aware AutoEncoder (LAE) to bridge the generalization gap. In the training process, we use a pixel-wise mask to regularize local interpretation of LAE to enforce the model to learn intrinsic representation from the forgery region, instead of capturing artifacts in the training set and learning superficial correlations to perform detection. We further propose an active learning framework to select the challenging candidates for labeling, which requires human masks for less than 3% of the training data, dramatically reducing the annotation efforts to regularize interpretations. Experimental results on three deepfake detection tasks indicate that LAE could focus on the forgery regions to make decisions. The analysis further shows that LAE outperforms the state-of-the-arts by 6.52%, 12.03%, and 3.08% respectively on three deepfake detection tasks in terms of generalization accuracy on previously unseen manipulations.

Key findings

LAE significantly improves generalization accuracy on unseen manipulations, outperforming state-of-the-art baselines by 6.52%, 12.03%, and 3.08% on three deepfake detection tasks. The method demonstrates that regularizing local interpretation with a small fraction of pixel-wise masks (less than 3% of training data selected via active learning) enables the model to focus on true forgery regions, bridging the generalization gap.

Approach

The proposed Locality-Aware AutoEncoder (LAE) uses an autoencoder-based architecture to learn fine-grained representations. It augments local interpretability and uses pixel-wise forgery ground truth masks to explicitly regularize the model's attention, forcing it to focus on manipulated regions. An active learning framework selects challenging candidates for mask annotation, minimizing the human labeling effort required.

Datasets

Faceforensics++ (Face2face, FaceSwap), CelebA (StarGAN, Glow), G&L, ContextAtten

Model(s)

Locality-Aware AutoEncoder (LAE) with a U-net like encoder-decoder structure. It integrates a VGGNet (16-layer version) for perceptual loss and a DCGAN discriminator for adversarial loss. (Note: The paper focuses on visual deepfake detection, not audio deepfake detection).

Author countries

USA

← Previous