Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors

Authors: Hasam Khalid, Minha Kim, Shahroz Tariq, Simon S. Woo

Published: 2021-09-07 11:00:20+00:00

Comment: 2 Figures, 2 Tables, Accepted for publication at the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection (ADGD '21) at ACM MM 2021

AI Summary

This paper evaluates the Audio-Video Multimodal Deepfake Detection Dataset (FakeAVCeleb) using state-of-the-art unimodal, ensemble-based, and multimodal detection methods. The study reveals that unimodal detectors, addressing only a single modality (audio or video), perform poorly compared to ensemble methods. Interestingly, purely multimodal baselines provided the worst detection performance on this dataset.

Abstract

Significant advancements made in the generation of deepfakes have caused security and privacy issues. Attackers can easily impersonate a person's identity in an image by replacing his face with the target person's face. Moreover, a new domain of cloning human voices using deep-learning technologies is also emerging. Now, an attacker can generate realistic cloned voices of humans using only a few seconds of audio of the target person. With the emerging threat of potential harm deepfakes can cause, researchers have proposed deepfake detection methods. However, they only focus on detecting a single modality, i.e., either video or audio. On the other hand, to develop a good deepfake detector that can cope with the recent advancements in deepfake generation, we need to have a detector that can detect deepfakes of multiple modalities, i.e., videos and audios. To build such a detector, we need a dataset that contains video and respective audio deepfakes. We were able to find a most recent deepfake dataset, Audio-Video Multimodal Deepfake Detection Dataset (FakeAVCeleb), that contains not only deepfake videos but synthesized fake audios as well. We used this multimodal deepfake dataset and performed detailed baseline experiments using state-of-the-art unimodal, ensemble-based, and multimodal detection methods to evaluate it. We conclude through detailed experimentation that unimodals, addressing only a single modality, video or audio, do not perform well compared to ensemble-based methods. Whereas purely multimodal-based baselines provide the worst performance.

Key findings

Unimodal detectors, focusing on either audio or video, do not perform as well as ensemble-based methods. Surprisingly, purely multimodal baseline methods performed the worst, indicating current multimodal approaches may not be suitable for this type of deepfake detection. Ensemble methods showed the best performance, but their accuracy remained below 85%, suggesting that multimodal deepfake detection is a complex task requiring further research.

Approach

The authors evaluate the FakeAVCeleb multimodal dataset by conducting baseline experiments using three categories of detection methods: unimodal (video-only or audio-only), ensemble-based (combining unimodal audio and video classifiers), and existing multimodal methods. They preprocess video frames and extract Mel-Frequency Cepstral Coefficients (MFCCs) from audio for input to various deep learning models.

Datasets

Audio-Video Multimodal Deepfake Detection Dataset (FakeAVCeleb), VoxCeleb2 (used to generate FakeAVCeleb)

Model(s)

VGG16, Meso-4, MesoInception-4, Xception, EfficientNet-B0, Multimodal-1 (modified), Multimodal-2 (modified), CDCN (modified)

Author countries

South Korea

← Previous