C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection

View on arXiv ← Back to list

Authors: Chuangchuang Tan, Renshuai Tao, Huan Liu, Guanghua Gu, Baoyuan Wu, Yao Zhao, Yunchao Wei

Published: 2024-08-19 02:14:25+00:00

AI Summary

This paper proposes C2P-CLIP, a method to improve the generalization of CLIP for deepfake detection. It achieves this by injecting category-related concepts into CLIP's image encoder using category common prompts, resulting in a significant accuracy improvement without adding parameters during testing.

Abstract

This work focuses on AIGC detection to develop universal detectors capable of identifying various types of forgery images. Recent studies have found large pre-trained models, such as CLIP, are effective for generalizable deepfake detection along with linear classifiers. However, two critical issues remain unresolved: 1) understanding why CLIP features are effective on deepfake detection through a linear classifier; and 2) exploring the detection potential of CLIP. In this study, we delve into the underlying mechanisms of CLIP's detection capabilities by decoding its detection features into text and performing word frequency analysis. Our finding indicates that CLIP detects deepfakes by recognizing similar concepts (Fig. ref{fig:fig1} a). Building on this insight, we introduce Category Common Prompt CLIP, called C2P-CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder, thereby enhancing detection performance (Fig. ref{fig:fig1} b). Our method achieves a 12.41% improvement in detection accuracy compared to the original CLIP, without introducing additional parameters during testing. Comprehensive experiments conducted on two widely-used datasets, encompassing 20 generation models, validate the efficacy of the proposed method, demonstrating state-of-the-art performance. The code is available at url{https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection}

Key findings

C2P-CLIP significantly outperforms the original CLIP and other state-of-the-art methods on both datasets, achieving a 12.41% improvement in accuracy on UniversalFakeDetect and a substantial improvement on GenImage. The method achieves this improvement without adding parameters during testing.

Approach

C2P-CLIP injects category common prompts (e.g., "Deepfake", "Camera") into CLIP's text encoder to influence its image encoder. This injects category-related concepts, improving the model's ability to distinguish real from fake images via contrastive and classification losses during training.

Datasets

UniversalFakeDetect dataset (with ProGAN for training and various models for testing), GenImage dataset (with SDv1.4 for training and various diffusion models for testing)

Model(s)

CLIP (ViT-L/14) with LoRA layers added to the image encoder during training.

Author countries

China

← Previous