DF-TransFusion: Multimodal Deepfake Detection via Lip-Audio Cross-Attention and Facial Self-Attention

View on arXiv ← Back to list

Authors: Aaditya Kharel, Manas Paranjape, Aniket Bera

Published: 2023-09-12 18:37:05+00:00

AI Summary

This paper introduces DF-TransFusion, a multimodal deepfake detection framework that processes audio and video concurrently. It utilizes cross-attention for lip synchronization analysis and self-attention for detecting facial artifacts, achieving state-of-the-art performance on multimodal deepfake detection.

Abstract

With the rise in manipulated media, deepfake detection has become an imperative task for preserving the authenticity of digital content. In this paper, we present a novel multi-modal audio-video framework designed to concurrently process audio and video inputs for deepfake detection tasks. Our model capitalizes on lip synchronization with input audio through a cross-attention mechanism while extracting visual cues via a fine-tuned VGG-16 network. Subsequently, a transformer encoder network is employed to perform facial self-attention. We conduct multiple ablation studies highlighting different strengths of our approach. Our multi-modal methodology outperforms state-of-the-art multi-modal deepfake detection techniques in terms of F-1 and per-video AUC scores.

Key findings

DF-TransFusion outperforms state-of-the-art multimodal deepfake detection methods in terms of F1 and per-video AUC scores on the DFDC and DF-TIMIT datasets. Ablation studies demonstrate the contributions of both the cross-attention and self-attention mechanisms.

Approach

DF-TransFusion employs a fine-tuned VGG-16 network for visual feature extraction and transformer encoders for both self-attention (on facial regions) and cross-attention (between lip movements and audio). The outputs are combined in a multi-layer perceptron for final classification.

Datasets

DFDC, DF-TIMIT, FakeAVCeleb

Model(s)

Fine-tuned VGG-16, Transformer encoder networks (with self-attention and cross-attention mechanisms), Multi-layer Perceptron (MLP)

Author countries

USA

← Previous