Finding Facial Forgery Artifacts with Parts-Based Detectors

View on arXiv ← Back to list

Authors: Steven Schwarcz, Rama Chellappa

Published: 2021-09-21 16:18:45+00:00

AI Summary

This paper proposes a novel parts-based approach for deepfake detection, dividing the face into regions (nose, mouth, eyes, chin) and training separate classifiers for each. This approach improves generalizability across datasets and provides insights into the forgery process.

Abstract

Manipulated videos, especially those where the identity of an individual has been modified using deep neural networks, are becoming an increasingly relevant threat in the modern day. In this paper, we seek to develop a generalizable, explainable solution to detecting these manipulated videos. To achieve this, we design a series of forgery detection systems that each focus on one individual part of the face. These parts-based detection systems, which can be combined and used together in a single architecture, meet all of our desired criteria - they generalize effectively between datasets and give us valuable insights into what the network is looking at when making its decision. We thus use these detectors to perform detailed empirical analysis on the FaceForensics++, Celeb-DF, and Facebook Deepfake Detection Challenge datasets, examining not just what the detectors find but also collecting and analyzing useful related statistics on the datasets themselves.

Key findings

Certain parts-based detectors (eyes and chin) outperform full-image baselines in cross-dataset generalization. The performance of parts-based detectors correlates with the distribution of artifacts in specific facial regions. Combining parts-based detectors doesn't always significantly improve performance.

Approach

The authors train separate convolutional neural networks (truncated Xception) on individual facial regions to detect deepfakes. These parts-based detectors, which can be combined into a single architecture, are evaluated on their generalizability across different deepfake datasets and algorithms.

Datasets

FaceForensics++, Celeb-DF, Facebook Deepfake Detection Challenge

Model(s)

Truncated Xception network

Author countries

USA

← Previous