Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes

View on arXiv ← Back to list

Authors: Kaiqing Lin, Zhiyuan Yan, Ke-Yue Zhang, Li Hao, Yue Zhou, Yuzhen Lin, Weixiang Li, Taiping Yao, Shouhong Ding, Bin Li

Published: 2025-05-26 06:55:23+00:00

AI Summary

VIPGuard is a multimodal deepfake detection framework that leverages prior knowledge of specific individuals' facial identities for improved accuracy and explainability. It fine-tunes a multimodal large language model (MLLM) to learn detailed facial attributes, performs identity-level discriminative learning, and incorporates user-specific customization for personalized detection.

Abstract

Securing personal identity against deepfake attacks is increasingly critical in the digital age, especially for celebrities and political figures whose faces are easily accessible and frequently targeted. Most existing deepfake detection methods focus on general-purpose scenarios and often ignore the valuable prior knowledge of known facial identities, e.g., VIP individuals whose authentic facial data are already available. In this paper, we propose textbf{VIPGuard}, a unified multimodal framework designed to capture fine-grained and comprehensive facial representations of a given identity, compare them against potentially fake or similar-looking faces, and reason over these comparisons to make accurate and explainable predictions. Specifically, our framework consists of three main stages. First, fine-tune a multimodal large language model (MLLM) to learn detailed and structural facial attributes. Second, we perform identity-level discriminative learning to enable the model to distinguish subtle differences between highly similar faces, including real and fake variations. Finally, we introduce user-specific customization, where we model the unique characteristics of the target face identity and perform semantic reasoning via MLLM to enable personalized and explainable deepfake detection. Our framework shows clear advantages over previous detection works, where traditional detectors mainly rely on low-level visual cues and provide no human-understandable explanations, while other MLLM-based models often lack a detailed understanding of specific face identities. To facilitate the evaluation of our method, we built a comprehensive identity-aware benchmark called textbf{VIPBench} for personalized deepfake detection, involving the latest 7 face-swapping and 7 entire face synthesis techniques for generation.

Key findings

VIPGuard significantly outperforms existing methods in deepfake detection, achieving near-perfect results on the proposed VIPBench dataset. It also demonstrates superior explainability by identifying subtle inconsistencies in local facial attributes. The method is effective even with limited real images per identity.

Approach

VIPGuard uses a three-stage approach: First, it fine-tunes an MLLM on a facial attribute dataset. Second, it performs identity-level discriminative learning using paired images. Finally, it incorporates a user-specific token for personalized detection and explanation.

Datasets

LAION-Face, FaceID-6M, CrossID; VIPBench (a new benchmark dataset with 22 identities and 80,080 images, including real and forged samples generated using 14 state-of-the-art methods); CelebDF, DF40

Model(s)

Multimodal Large Language Model (MLLM), specifically Qwen-2.5-VL-7B; pre-trained face models for extracting global and local facial priors; Gemini 2.5 Pro

Author countries

China

← Previous