C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection

Authors: Chuangchuang Tan, Renshuai Tao, Huan Liu, Guanghua Gu, Baoyuan Wu, Yao Zhao, Yunchao Wei

Published: 2024-08-19 02:14:25+00:00

Comment: 10 pages, 5 figures

AI Summary

This work explores why CLIP features are effective for deepfake detection via a linear classifier, revealing that CLIP identifies deepfakes by recognizing similar concepts. Building on this, the authors introduce C2P-CLIP, which injects category-related concepts into CLIP's image encoder through a category common prompt in the text encoder. This method significantly enhances deepfake detection generalization and achieves state-of-the-art performance without adding test-time parameters.

Abstract

This work focuses on AIGC detection to develop universal detectors capable of identifying various types of forgery images. Recent studies have found large pre-trained models, such as CLIP, are effective for generalizable deepfake detection along with linear classifiers. However, two critical issues remain unresolved: 1) understanding why CLIP features are effective on deepfake detection through a linear classifier; and 2) exploring the detection potential of CLIP. In this study, we delve into the underlying mechanisms of CLIP's detection capabilities by decoding its detection features into text and performing word frequency analysis. Our finding indicates that CLIP detects deepfakes by recognizing similar concepts (Fig. \\ref{fig:fig1} a). Building on this insight, we introduce Category Common Prompt CLIP, called C2P-CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder, thereby enhancing detection performance (Fig. \\ref{fig:fig1} b). Our method achieves a 12.41\\% improvement in detection accuracy compared to the original CLIP, without introducing additional parameters during testing. Comprehensive experiments conducted on two widely-used datasets, encompassing 20 generation models, validate the efficacy of the proposed method, demonstrating state-of-the-art performance. The code is available at \\url{https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection}

Key findings

The study found that CLIP detects deepfakes by recognizing similar concepts, rather than discerning true/false semantics. C2P-CLIP achieved a 12.41% improvement in detection accuracy and 8.52% in mAP compared to the original CLIP (UniFD baseline) on the UniversalFakeDetect dataset. The proposed method also demonstrated state-of-the-art performance, outperforming recent methods like FatFormer by 2.93% in accuracy on UniversalFakeDetect and improving accuracy by 7.0% over UniFD on the GenImage dataset.

Approach

The authors first analyze CLIP's detection capabilities by decoding detection features into text, concluding that CLIP detects deepfakes by matching similar concepts. They then propose C2P-CLIP, which integrates a 'category common prompt' (e.g., 'Deepfake' or 'Camera') into the text encoder to inject category-related concepts into the image encoder. This is achieved by fine-tuning the CLIP image encoder with LoRA using contrastive and classification losses, without introducing additional parameters during testing.

Datasets

UniversalFakeDetect dataset, GenImage dataset

Model(s)

CLIP (ViT-L/14), LoRA (Low-Rank Adaptation), Linear Classifier

Author countries

China

← Previous