SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework

View on arXiv ← Back to list

Authors: Binh M. Le, Jiwon Kim, Simon S. Woo, Kristen Moore, Alsharif Abuadbba, Shahroz Tariq

Published: 2024-01-09 05:32:22+00:00

AI Summary

This paper systematically reviews and analyzes state-of-the-art deepfake detectors, proposing a unified conceptual framework to categorize them based on 18 key factors. It then evaluates the generalizability of 16 leading detectors across black-box, gray-box, and white-box settings, providing insights into their performance and limitations.

Abstract

Deepfakes have rapidly emerged as a serious threat to society due to their ease of creation and dissemination, triggering the accelerated development of detection technologies. However, many existing detectors rely on labgenerated datasets for validation, which may not prepare them for novel, real-world deepfakes. This paper extensively reviews and analyzes state-of-the-art deepfake detectors, evaluating them against several critical criteria. These criteria categorize detectors into 4 high-level groups and 13 finegrained sub-groups, aligned with a unified conceptual framework we propose. This classification offers practical insights into the factors affecting detector efficacy. We evaluate the generalizability of 16 leading detectors across comprehensive attack scenarios, including black-box, white-box, and graybox settings. Our systematized analysis and experiments provide a deeper understanding of deepfake detectors and their generalizability, paving the way for future research and the development of more proactive defenses against deepfakes.

Key findings

Spatiotemporal models generally performed best, but no single detector consistently excelled across all evaluation settings. The study highlighted the importance of considering multiple factors in detector design and the limitations of generalizability claims based solely on gray-box evaluations. Attention-based and multiple-frame-based approaches showed promise, especially in black-box settings.

Approach

The authors propose a five-step conceptual framework for categorizing deepfake detectors based on 18 influential factors. They evaluate 16 leading detectors across three evaluation settings (black-box, gray-box, and white-box) using various datasets to assess their generalizability and identify the impact of influential factors on detector performance.

Datasets

FaceForensics++, DeepFake Detection Challenge (DFDC), Celebrity Deepfake (CelebDF), Audio-Video Deepfake (FakeAVCeleb), a novel "white-box" deepfake dataset created by the authors, and a black-box dataset of 2000 in-the-wild samples from Reddit, YouTube, Bilibili, and TikTok.

Model(s)

Various deep neural networks including ConvNets (e.g., VGG, ResNet, XceptionNet), sequence models (e.g., BiLSTM), and specialized networks (e.g., graph learning, capsule networks). Specific models evaluated include Cap.Forensics, XceptionNet, MAT, ADD, LipForensics, EffB4Att, FTCN, MCX-API, AltFreezing, CADDM, SBIs, CCViT, ICT, LRNet, and LGrad.

Author countries

South Korea, Australia

← Previous