Integrating Audio-Visual Features for Multimodal Deepfake Detection

View on arXiv ← Back to list

Authors: Sneha Muppalla, Shan Jia, Siwei Lyu

Published: 2023-10-05 18:19:56+00:00

AI Summary

This paper proposes a novel audio-visual deepfake detection method that integrates fine-grained deepfake identification with binary classification. By categorizing samples into four types based on audio and video authenticity, it improves detection accuracy across intra-domain and cross-domain testing.

Abstract

Deepfakes are AI-generated media in which an image or video has been digitally modified. The advancements made in deepfake technology have led to privacy and security issues. Most deepfake detection techniques rely on the detection of a single modality. Existing methods for audio-visual detection do not always surpass that of the analysis based on single modalities. Therefore, this paper proposes an audio-visual-based method for deepfake detection, which integrates fine-grained deepfake identification with binary classification. We categorize the samples into four types by combining labels specific to each single modality. This method enhances the detection under intra-domain and cross-domain testing.

Key findings

The proposed method outperforms existing single-modality and audio-visual detection methods on both FakeAVCeleb and TMC datasets, achieving higher accuracy and AUC scores in intra-domain and cross-domain testing. The Swin Transformer model demonstrated superior generalization ability compared to the Capsule Network.

Approach

The approach uses a multi-task learning strategy that combines three loss functions: binary cross-entropy losses for audio and video modalities and a four-class cross-entropy loss for fine-grained deepfake identification. Audio and video features are extracted using deep neural networks (Capsule Network and Swin Transformer are used in experiments), then fused (either feature or score fusion) before classification.

Datasets

FakeAVCeleb and TMC datasets

Model(s)

Capsule Network and Swin Transformer

Author countries

USA

← Previous