Quantitative Metrics for Evaluating Explanations of Video DeepFake Detectors

Authors: Federico Baldassarre, Quentin Debard, Gonzalo Fiz Pontiveros, Tri Kurniawan Wijaya

Published: 2022-10-07 16:41:46+00:00

Comment: Accepted at BMVC 2022, code repository at https://github.com/baldassarreFe/deepfake-detection

AI Summary

This paper introduces a set of quantitative, human-centric metrics to evaluate the visual quality and informativeness of explanations for video DeepFake classifiers. It aims to address the reliance on qualitative comparisons by providing an objective framework for assessing properties like smoothness, locality, sparsity, and manipulation detection. The authors apply these metrics to compare various techniques for improving explanation quality and performance on benchmark DeepFake datasets.

Abstract

The proliferation of DeepFake technology is a rising challenge in today's society, owing to more powerful and accessible generation methods. To counter this, the research community has developed detectors of ever-increasing accuracy. However, the ability to explain the decisions of such models to users is lacking behind and is considered an accessory in large-scale benchmarks, despite being a crucial requirement for the correct deployment of automated tools for content moderation. We attribute the issue to the reliance on qualitative comparisons and the lack of established metrics. We describe a simple set of metrics to evaluate the visual quality and informativeness of explanations of video DeepFake classifiers from a human-centric perspective. With these metrics, we compare common approaches to improve explanation quality and discuss their effect on both classification and explanation performance on the recent DFDC and DFD datasets.

Key findings

Total Variation regularization significantly enhanced explanation quality across most metrics, improving locality, sparsity, and manipulation detection. The Multi-scale Vision Transformer (MViT) architecture notably outperformed S3D CNNs in both DeepFake detection accuracy and explanation quality, yielding smoother, sparser, and better manipulation-detecting heatmaps. Preprocessing with high-frequency smoothing filters did not consistently improve video DeepFake explanations.

Approach

The authors propose quantitative metrics to evaluate heatmap-based explanations of video DeepFake classifiers from a human-centric perspective. These metrics assess visual quality (smoothness, spatial locality, sparsity) and manipulation detection accuracy against ground-truth masks. They then apply this framework to evaluate the impact of data preparation, regularization, augmentation, and model architecture on explanation quality.

Datasets

DeepFake Detection Challenge (DFDC), DeepFake Detection Dataset (DFD)

Model(s)

S3D (3D CNN), Multi-scale Vision Transformer (MViT)

Author countries

Sweden, Ireland

← Previous