Robust CLIP-Based Detector for Exposing Diffusion Model-Generated Images

View on arXiv ← Back to list

Authors: Santosh, Li Lin, Irene Amerini, Xin Wang, Shu Hu

Published: 2024-04-19 14:30:41+00:00

AI Summary

This paper proposes a robust image deepfake detection framework that integrates image and text features extracted by CLIP with an MLP classifier. A novel loss function improves robustness and handles imbalanced datasets, while loss landscape flattening enhances generalization.

Abstract

Diffusion models (DMs) have revolutionized image generation, producing high-quality images with applications spanning various fields. However, their ability to create hyper-realistic images poses significant challenges in distinguishing between real and synthetic content, raising concerns about digital authenticity and potential misuse in creating deepfakes. This work introduces a robust detection framework that integrates image and text features extracted by CLIP model with a Multilayer Perceptron (MLP) classifier. We propose a novel loss that can improve the detector's robustness and handle imbalanced datasets. Additionally, we flatten the loss landscape during the model training to improve the detector's generalization capabilities. The effectiveness of our method, which outperforms traditional detection techniques, is demonstrated through extensive experiments, underscoring its potential to set a new state-of-the-art approach in DM-generated image detection. The code is available at https://github.com/Purdue-M2/Robust_DM_Generated_Image_Detection.

Key findings

The proposed method achieves a near-perfect AUC score of 99.999854%, outperforming baseline methods. Ablation studies demonstrate the effectiveness of the CVaR, AUC losses, and SAM optimization. The sensitivity analysis identifies optimal hyperparameters for the CVaR and AUC losses.

Approach

The approach uses CLIP to extract image and text features, concatenates them, and feeds them into a three-layer MLP classifier. A novel loss function combining CVaR and AUC losses is used to improve robustness and handle class imbalance, and SAM is employed for loss landscape flattening.

Datasets

Diffusion-generated Deepfake Detection dataset (D3), which includes real images from LAION-400M and synthetic images generated by Stable Diffusion 1.4, Stable Diffusion 2.1, Stable Diffusion XL, and DeepFloyd IF.

Model(s)

CLIP (for feature extraction), Multilayer Perceptron (MLP) classifier

Author countries

USA, Italy

← Previous