Exploiting Style Latent Flows for Generalizing Deepfake Video Detection

Authors: Jongwook Choi, Taehoon Kim, Yonghyun Jeong, Seungryul Baek, Jongwon Choi

Published: 2024-03-11 10:35:58+00:00

Comment: Preprint version, final version will be available at https://openaccess.thecvf.com The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR) (2024) Published by: IEEE & CVF

AI Summary

This paper introduces a novel deepfake video detection approach that exploits the abnormal temporal changes in style latent vectors, which are distinct in generated facial videos. The framework utilizes a StyleGRU module, trained with contrastive learning, to capture these dynamic properties, and a style attention module to integrate these features with content-based features. The method demonstrates state-of-the-art performance in cross-dataset and cross-manipulation scenarios, highlighting the importance of style latent flows for generalization.

Abstract

This paper presents a new approach for the detection of fake videos, based on the analysis of style latent vectors and their abnormal behavior in temporal changes in the generated videos. We discovered that the generated facial videos suffer from the temporal distinctiveness in the temporal changes of style latent vectors, which are inevitable during the generation of temporally stable videos with various facial expressions and geometric transformations. Our framework utilizes the StyleGRU module, trained by contrastive learning, to represent the dynamic properties of style latent vectors. Additionally, we introduce a style attention module that integrates StyleGRU-generated features with content-based features, enabling the detection of visual and temporal artifacts. We demonstrate our approach across various benchmark scenarios in deepfake detection, showing its superiority in cross-dataset and cross-manipulation scenarios. Through further analysis, we also validate the importance of using temporal changes of style latent vectors to improve the generality of deepfake video detection.

Key findings

The research demonstrates that temporal changes in style latent vectors provide an effective and generalizable cue for deepfake video detection. Their proposed framework, integrating StyleGRU and a style attention module, achieves state-of-the-art performance in challenging cross-dataset and cross-manipulation scenarios. This confirms the approach's robustness against diverse deepfake generation methods and validates the importance of style latent flows for improved generality.

Approach

The method detects deepfake videos by analyzing the temporal variations (style latent flows) of style latent vectors extracted from consecutive video frames. It employs a StyleGRU module, trained with supervised contrastive learning, to capture these dynamics, which are then integrated with content-based features via a Style Attention Module for robust detection and classification by a Temporal Transformer Encoder.

Datasets

FaceForensics++ (FF++), FaceShifter (FSh), DeeperForensics (DFo), CelebDF-v2 (CDF), DeepfakeDetection (DFD), DFDC

Model(s)

pSp encoder, StyleGRU (Gated Recurrent Unit), 3D ResNet-50, Style Attention Module (SAM), Temporal Transformer Encoder (TTE), Multi-Layer Perceptron (MLP)

Author countries

Korea

← Previous