Generalize Your Face Forgery Detectors: An Insertable Adaptation Module Is All You Need

View on arXiv ← Back to list

Authors: Xiaotian Si, Linghui Li, Liwei Zhang, Ziduo Guo, Kaiguo Yuan, Bingyu Li, Xiaoyong Li

Published: 2024-12-30 08:48:04+00:00

AI Summary

This paper introduces an insertable adaptation module for face forgery detectors that enhances generalization to unseen forgeries. The module adapts a pre-trained detector using unlabeled test data, improving performance without architectural modifications. This is achieved through a learnable class prototype-based classifier and a nearest feature calibrator.

Abstract

A plethora of face forgery detectors exist to tackle facial deepfake risks. However, their practical application is hindered by the challenge of generalizing to forgeries unseen during the training stage. To this end, we introduce an insertable adaptation module that can adapt a trained off-the-shelf detector using only online unlabeled test data, without requiring modifications to the architecture or training process. Specifically, we first present a learnable class prototype-based classifier that generates predictions from the revised features and prototypes, enabling effective handling of various forgery clues and domain gaps during online testing. Additionally, we propose a nearest feature calibrator to further improve prediction accuracy and reduce the impact of noisy pseudo-labels during self-training. Experiments across multiple datasets show that our module achieves superior generalization compared to state-of-the-art methods. Moreover, it functions as a plug-and-play component that can be combined with various detectors to enhance the overall performance.

Key findings

The adaptation module significantly outperforms state-of-the-art methods in generalization across multiple datasets. It improves the performance of various base detectors, demonstrating its plug-and-play capability. Ablation studies confirm the effectiveness of both the classifier and calibrator components.

Approach

The proposed method uses an insertable adaptation module that includes a learnable class prototype-based classifier and a nearest feature calibrator. The classifier generates predictions from revised features and prototypes, handling forgery clues and domain gaps. The calibrator improves accuracy and reduces noisy pseudo-labels.

Datasets

FaceForensics++ (FF++) (c23 version), DeepfakeDetection (DFD), Deepfake Detection Challenge (DFDC), CelebDF (CDF)

Model(s)

CLIP, Xception, Recce, SRM, Effinb4 (as base detectors; the proposed module is used with various pre-trained models)

Author countries

China

← Previous