Interpretable and Trustworthy Deepfake Detection via Dynamic Prototypes

View on arXiv ← Back to list

Authors: Loc Trinh, Michael Tsang, Sirisha Rambhatla, Yan Liu

Published: 2020-06-28 00:25:34+00:00

AI Summary

This paper introduces DPNet, a novel deepfake detection method that leverages dynamic prototypes to explain temporal inconsistencies in videos. DPNet achieves competitive performance on various datasets while providing human-understandable visual explanations of its predictions, enhancing trustworthiness.

Abstract

In this paper we propose a novel human-centered approach for detecting forgery in face images, using dynamic prototypes as a form of visual explanations. Currently, most state-of-the-art deepfake detections are based on black-box models that process videos frame-by-frame for inference, and few closely examine their temporal inconsistencies. However, the existence of such temporal artifacts within deepfake videos is key in detecting and explaining deepfakes to a supervising human. To this end, we propose Dynamic Prototype Network (DPNet) -- an interpretable and effective solution that utilizes dynamic representations (i.e., prototypes) to explain deepfake temporal artifacts. Extensive experimental results show that DPNet achieves competitive predictive performance, even on unseen testing datasets such as Google's DeepFakeDetection, DeeperForensics, and Celeb-DF, while providing easy referential explanations of deepfake dynamics. On top of DPNet's prototypical framework, we further formulate temporal logic specifications based on these dynamics to check our model's compliance to desired temporal behaviors, hence providing trustworthiness for such critical detection systems.

Key findings

DPNet achieves state-of-the-art or competitive performance on various deepfake detection datasets, including unseen ones. The model provides interpretable visual explanations via dynamic prototypes, improving trustworthiness. Temporal logic specifications further enhance the model's verifiability and robustness.

Approach

DPNet uses a prototype-based neural network to learn representations of temporal inconsistencies in deepfake videos. It makes predictions based on the similarity between test video dynamics and learned dynamic prototypes, which are then projected to the closest representative video patches for visual explanation.

Datasets

FaceForensics++, DeepFakeDetection, DeeperForensics-1.0, Celeb-DF

Model(s)

Dynamic Prototype Network (DPNet) using HRNet as a feature encoder backbone.

Author countries

USA

← Previous