Interpretable and Trustworthy Deepfake Detection via Dynamic Prototypes

Authors: Loc Trinh, Michael Tsang, Sirisha Rambhatla, Yan Liu

Published: 2020-06-28 00:25:34+00:00

Comment: To appear in the 2021 IEEE Winter Conference on Applications of Computer Vision (WACV 21')

AI Summary

This paper introduces the Dynamic Prototype Network (DPNet), a novel human-centered approach for interpretable deepfake detection in face images within videos. DPNet utilizes dynamic prototypes to identify and explain temporal inconsistencies and artifacts, crucial for detecting sophisticated deepfakes. It achieves competitive predictive performance on various unseen datasets and enhances trustworthiness by formulating temporal logic specifications based on learned dynamics.

Abstract

In this paper we propose a novel human-centered approach for detecting forgery in face images, using dynamic prototypes as a form of visual explanations. Currently, most state-of-the-art deepfake detections are based on black-box models that process videos frame-by-frame for inference, and few closely examine their temporal inconsistencies. However, the existence of such temporal artifacts within deepfake videos is key in detecting and explaining deepfakes to a supervising human. To this end, we propose Dynamic Prototype Network (DPNet) -- an interpretable and effective solution that utilizes dynamic representations (i.e., prototypes) to explain deepfake temporal artifacts. Extensive experimental results show that DPNet achieves competitive predictive performance, even on unseen testing datasets such as Google's DeepFakeDetection, DeeperForensics, and Celeb-DF, while providing easy referential explanations of deepfake dynamics. On top of DPNet's prototypical framework, we further formulate temporal logic specifications based on these dynamics to check our model's compliance to desired temporal behaviors, hence providing trustworthiness for such critical detection systems.

Key findings

DPNet achieved superior predictive performance on FF++ (e.g., 99.20% AUC on HQ c23) and demonstrated strong generalization to unseen datasets like DeepFakeDetection, DeeperForensics-1.0, and Celeb-DF, improving AUC by 1-4% over state-of-the-art baselines. The approach provides human-understandable visual explanations of temporal artifacts via dynamic prototypes and allows for robust trustworthiness checks using temporal logic specifications.

Approach

DPNet processes video inputs by stacking RGB frames with precomputed optical flow fields using an HRNet backbone. It learns dynamic prototype vectors representing temporal artifact patterns in the latent space, making predictions based on the similarity between a test video's dynamics and these prototypes. The framework also incorporates temporal logic specifications to verify model compliance with desired temporal behaviors, ensuring trustworthiness.

Datasets

FaceForensics++ (FF++), DeepFakeDetection (DFD), DeeperForensics-1.0, Celeb-DF

Model(s)

Dynamic Prototype Network (DPNet) with HRNet as a backbone feature encoder for video deepfake detection.

Author countries

United States

← Previous