NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake Detection

Authors: Yu Chen, Yang Yu, Rongrong Ni, Yao Zhao, Haoliang Li

Published: 2023-06-12 06:06:05+00:00

AI Summary

This paper introduces NPVForensics, a novel Deepfake detection method that exploits the correlation between non-critical phonemes and visemes, which are often overlooked by advanced Deepfake manipulation techniques. The approach utilizes a Local Feature Aggregation Swin Transformer (LFA-ST) for subtle feature extraction, an evolutionary consistency loss for fine-grained motion analysis, and a Phoneme-Viseme Awareness Module (PVAM) for cross-modal fusion and alignment. A self-supervised pre-training strategy on natural videos enables the model to adapt effectively to downstream Deepfake detection tasks.

Abstract

Deepfake technologies empowered by deep learning are rapidly evolving, creating new security concerns for society. Existing multimodal detection methods usually capture audio-visual inconsistencies to expose Deepfake videos. More seriously, the advanced Deepfake technology realizes the audio-visual calibration of the critical phoneme-viseme regions, achieving a more realistic tampering effect, which brings new challenges. To address this problem, we propose a novel Deepfake detection method to mine the correlation between Non-critical Phonemes and Visemes, termed NPVForensics. Firstly, we propose the Local Feature Aggregation block with Swin Transformer (LFA-ST) to construct non-critical phoneme-viseme and corresponding facial feature streams effectively. Secondly, we design a loss function for the fine-grained motion of the talking face to measure the evolutionary consistency of non-critical phoneme-viseme. Next, we design a phoneme-viseme awareness module for cross-modal feature fusion and representation alignment, so that the modality gap can be reduced and the intrinsic complementarity of the two modalities can be better explored. Finally, a self-supervised pre-training strategy is leveraged to thoroughly learn the audio-visual correspondences in natural videos. In this manner, our model can be easily adapted to the downstream Deepfake datasets with fine-tuning. Extensive experiments on existing benchmarks demonstrate that the proposed approach outperforms state-of-the-art methods.

Key findings

NPVForensics significantly outperforms state-of-the-art methods in Deepfake detection across various compression rates on FF++ and demonstrates superior cross-manipulation generalization. The model exhibits strong generalization ability across diverse Deepfake datasets, including challenging lip-sync Deepfakes, achieving an average AUC of 92.5% on four unseen datasets and 90.1% on lip-sync datasets. Furthermore, the self-supervised pre-training strategy on large-scale real videos greatly enhances the model's robustness to common video degradations and improves detection accuracy.

Approach

NPVForensics extracts non-critical phoneme and viseme features using an LFA-ST backbone, designed to capture subtle local and global information. It then calculates an Evolutionary Consistency Loss to measure fine-grained audio-visual inconsistencies and employs a Phoneme-Viseme Awareness Module (PVAM) for cross-modal feature fusion and representation alignment, reducing the modality gap. The entire framework benefits from a self-supervised pre-training strategy on large-scale real videos before fine-tuning on Deepfake datasets.

Datasets

VoxCeleb2, AV Speech dataset, FaceForensics++ (FF++), FaceShifter (FSh), Celeb-DF-v2, DeeperForensics (DFo), Deepfake Detection Challenge Dataset (DFDC), FakeAVCeleb, Audio-to-Video (A2V), Text-to-Video (T2V-S), Text-to-Video (T2V-L)

Model(s)

Swin Transformer, Local Feature Aggregation (LFA) block, Multi-modal Projection Head, Phoneme-Viseme Awareness Module (PVAM), Cross Attentional Fusion Module (CAFM), Co-correlation Guided Representation Alignment (CGRA)

Author countries

China

← Previous