Detect Any Deepfakes: Segment Anything Meets Face Forgery Detection and Localization

View on arXiv ← Back to list

Authors: Yingxin Lai, Zhiming Luo, Zitong Yu

Published: 2023-06-29 16:25:04+00:00

AI Summary

This paper introduces DADF, a deepfake detection and localization framework that leverages the Segment Anything Model (SAM). DADF incorporates a Multiscale Adapter for efficient fine-tuning and a Reconstruction Guided Attention module to enhance forgery detection sensitivity. Extensive experiments demonstrate state-of-the-art performance.

Abstract

The rapid advancements in computer vision have stimulated remarkable progress in face forgery techniques, capturing the dedicated attention of researchers committed to detecting forgeries and precisely localizing manipulated areas. Nonetheless, with limited fine-grained pixel-wise supervision labels, deepfake detection models perform unsatisfactorily on precise forgery detection and localization. To address this challenge, we introduce the well-trained vision segmentation foundation model, i.e., Segment Anything Model (SAM) in face forgery detection and localization. Based on SAM, we propose the Detect Any Deepfakes (DADF) framework with the Multiscale Adapter, which can capture short- and long-range forgery contexts for efficient fine-tuning. Moreover, to better identify forged traces and augment the model's sensitivity towards forgery regions, Reconstruction Guided Attention (RGA) module is proposed. The proposed framework seamlessly integrates end-to-end forgery localization and detection optimization. Extensive experiments on three benchmark datasets demonstrate the superiority of our approach for both forgery detection and localization. The codes will be released soon at https://github.com/laiyingxin2/DADF.

Key findings

DADF achieves state-of-the-art performance in both forgery detection and localization across multiple benchmark datasets. The Multiscale Adapter and RGA module significantly improve upon the baseline SAM performance. DADF shows strong generalization capabilities in cross-dataset testing.

Approach

The authors propose DADF, which integrates SAM with a Multiscale Adapter to capture short- and long-range forgery contexts. A Reconstruction Guided Attention (RGA) module is added to improve sensitivity to forged regions. The framework performs end-to-end forgery localization and detection optimization.

Datasets

FaceForensics++ (FF++) with different compression ratios (original, high quality, low quality), DF-TIMI, DFD, and FMLD.

Model(s)

Segment Anything Model (SAM) with added Multiscale Adapter and Reconstruction Guided Attention (RGA) module.

Author countries

China

← Previous