Quantitative Metrics for Evaluating Explanations of Video DeepFake Detectors

View on arXiv ← Back to list

Authors: Federico Baldassarre, Quentin Debard, Gonzalo Fiz Pontiveros, Tri Kurniawan Wijaya

Published: 2022-10-07 16:41:46+00:00

AI Summary

This paper introduces quantitative metrics to evaluate the visual quality and informativeness of explanations for video deepfake classifiers. These metrics, focusing on smoothness, sparsity, and locality, are used to compare different explanation approaches on the DFDC and DFD datasets, addressing the lack of established metrics for evaluating DeepFake explanation methods.

Abstract

The proliferation of DeepFake technology is a rising challenge in today's society, owing to more powerful and accessible generation methods. To counter this, the research community has developed detectors of ever-increasing accuracy. However, the ability to explain the decisions of such models to users is lacking behind and is considered an accessory in large-scale benchmarks, despite being a crucial requirement for the correct deployment of automated tools for content moderation. We attribute the issue to the reliance on qualitative comparisons and the lack of established metrics. We describe a simple set of metrics to evaluate the visual quality and informativeness of explanations of video DeepFake classifiers from a human-centric perspective. With these metrics, we compare common approaches to improve explanation quality and discuss their effect on both classification and explanation performance on the recent DFDC and DFD datasets.

Key findings

The proposed metrics effectively quantify the visual quality of explanations. The study compares different explanation techniques, providing a quantitative baseline for evaluating their performance on DeepFake detection datasets. The results highlight the importance of considering human-centric properties when evaluating the explainability of deepfake detectors.

Approach

The authors propose a set of quantitative metrics (smoothness, sparsity, locality) to evaluate the quality of heatmap-based explanations generated by video deepfake detectors. These metrics are applied to compare different explanation methods (e.g., SmoothGrad, Integrated Gradients) on their ability to produce human-interpretable explanations.

Datasets

DFDC and DFD datasets

Model(s)

SmoothGrad is used as a baseline for evaluation; other models are mentioned but not as the primary focus.

Author countries

Sweden, Ireland

← Previous