Towards Generalizable Deepfake Detection with Locality-aware AutoEncoder

View on arXiv ← Back to list

Authors: Mengnan Du, Shiva Pentyala, Yuening Li, Xia Hu

Published: 2019-09-13 02:01:32+00:00

AI Summary

This paper proposes Locality-Aware AutoEncoder (LAE) for generalizable deepfake detection, addressing the issue of existing methods' poor performance on unseen manipulations. LAE uses pixel-wise masks to regularize local interpretation, forcing the model to learn from forgery regions rather than superficial correlations, and incorporates an active learning framework to reduce annotation effort.

Abstract

With advancements of deep learning techniques, it is now possible to generate super-realistic images and videos, i.e., deepfakes. These deepfakes could reach mass audience and result in adverse impacts on our society. Although lots of efforts have been devoted to detect deepfakes, their performance drops significantly on previously unseen but related manipulations and the detection generalization capability remains a problem. Motivated by the fine-grained nature and spatial locality characteristics of deepfakes, we propose Locality-Aware AutoEncoder (LAE) to bridge the generalization gap. In the training process, we use a pixel-wise mask to regularize local interpretation of LAE to enforce the model to learn intrinsic representation from the forgery region, instead of capturing artifacts in the training set and learning superficial correlations to perform detection. We further propose an active learning framework to select the challenging candidates for labeling, which requires human masks for less than 3% of the training data, dramatically reducing the annotation efforts to regularize interpretations. Experimental results on three deepfake detection tasks indicate that LAE could focus on the forgery regions to make decisions. The analysis further shows that LAE outperforms the state-of-the-arts by 6.52%, 12.03%, and 3.08% respectively on three deepfake detection tasks in terms of generalization accuracy on previously unseen manipulations.

Key findings

LAE significantly outperforms state-of-the-art methods in generalization accuracy on unseen manipulations, achieving improvements of 6.52%, 12.03%, and 3.08% across three deepfake detection tasks. The active learning framework effectively reduces annotation effort, requiring masks for less than 3% of the training data. LAE's attention maps demonstrate a focus on forgery regions, unlike baseline methods.

Approach

The authors propose Locality-Aware AutoEncoder (LAE), which combines an autoencoder for representation learning with pixel-wise forgery masks to regularize local interpretation. An active learning framework is used to efficiently select samples for mask annotation, improving generalization.

Datasets

Faceforensics++, CelebA, datasets generated using Face2Face, FaceSwap, StarGAN, Glow, G&L, and ContextAtten methods for face swap, facial attribute manipulation, and inpainting tasks.

Model(s)

Locality-Aware AutoEncoder (LAE), a type of autoencoder with added components for local interpretability regularization and active learning. Baselines include SuppressNet, ResidualNet, StatsNet, MesoInception, XceptionNet, and ForensicTransfer.

Author countries

USA

← Previous