In Anticipation of Perfect Deepfake: Identity-anchored Artifact-agnostic Detection under Rebalanced Deepfake Detection Protocol

View on arXiv ← Back to list

Authors: Wei-Han Wang, Chin-Yuan Yeh, Hsi-Wen Chen, De-Nian Yang, Ming-Syan Chen

Published: 2024-05-01 12:48:13+00:00

AI Summary

This paper introduces the Rebalanced Deepfake Detection Protocol (RDDP) to evaluate deepfake detectors under balanced scenarios, mitigating the reliance on artifacts. It proposes ID-Miner, an artifact-agnostic detector that focuses on identity-anchored motion features for robust deepfake detection.

Abstract

As deep generative models advance, we anticipate deepfakes achieving perfection-generating no discernible artifacts or noise. However, current deepfake detectors, intentionally or inadvertently, rely on such artifacts for detection, as they are exclusive to deepfakes and absent in genuine examples. To bridge this gap, we introduce the Rebalanced Deepfake Detection Protocol (RDDP) to stress-test detectors under balanced scenarios where genuine and forged examples bear similar artifacts. We offer two RDDP variants: RDDP-WHITEHAT uses white-hat deepfake algorithms to create 'self-deepfakes,' genuine portrait videos with the resemblance of the underlying identity, yet carry similar artifacts to deepfake videos; RDDP-SURROGATE employs surrogate functions (e.g., Gaussian noise) to process both genuine and forged examples, introducing equivalent noise, thereby sidestepping the need of deepfake algorithms. Towards detecting perfect deepfake videos that aligns with genuine ones, we present ID-Miner, a detector that identifies the puppeteer behind the disguise by focusing on motion over artifacts or appearances. As an identity-based detector, it authenticates videos by comparing them with reference footage. Equipped with the artifact-agnostic loss at frame-level and the identity-anchored loss at video-level, ID-Miner effectively singles out identity signals amidst distracting variations. Extensive experiments comparing ID-Miner with 12 baseline detectors under both conventional and RDDP evaluations with two deepfake datasets, along with additional qualitative studies, affirm the superiority of our method and the necessity for detectors designed to counter perfect deepfakes.

Key findings

ID-Miner outperforms 12 baseline detectors under both conventional and RDDP evaluations on two datasets. The significant performance drop of baseline methods under RDDP highlights their reliance on artifacts. ID-Miner demonstrates robustness and generalizability across different deepfake techniques.

Approach

ID-Miner detects deepfakes by focusing on motion patterns instead of artifacts. It uses a pre-trained FAU extractor, an artifact-agnostic frame-level encoder, and an identity-anchored video-level aggregator with contrastive losses to learn identity-based features from action sequences.

Datasets

VoxCeleb and Celeb-DF

Model(s)

ID-Miner (using OpenFace for FAU extraction and GRUs for video-level aggregation), Xception, MesoNet, EfficientNet, FTCN, TI2Net, PWL, A&B, ID-Reveal

Author countries

Taiwan

← Previous