CAST: Cross-Attentive Spatio-Temporal feature fusion for Deepfake detection

Authors: Aryan Thakre, Omkar Nagwekar, Vedang Talekar, Aparna Santra Biswas

Published: 2025-06-26 18:51:17+00:00

Comment: 50 pages, 6 figures

AI Summary

CAST introduces a novel Cross-Attentive Spatio-Temporal feature fusion architecture for robust deepfake video detection, which leverages cross-attention to deeply integrate spatial and temporal features. This design allows temporal features to dynamically attend to relevant spatial regions, thereby enhancing the model's ability to detect subtle, time-evolving artifacts like flickering eyes or warped lips. The model demonstrates strong performance with an AUC of 99.49% and an accuracy of 97.57% in intra-dataset evaluations, and impressive generalization by achieving a 93.31% AUC on the unseen DeepfakeDetection dataset.

Abstract

Deepfakes have emerged as a significant threat to digital media authenticity, increasing the need for advanced detection techniques that can identify subtle and time-dependent manipulations. CNNs are effective at capturing spatial artifacts, and Transformers excel at modeling temporal inconsistencies. However, many existing CNN-Transformer models process spatial and temporal features independently. In particular, attention-based methods often use separate attention mechanisms for spatial and temporal features and combine them using naive approaches like averaging, addition, or concatenation, which limits the depth of spatio-temporal interaction. To address this challenge, we propose a unified CAST model that leverages cross-attention to effectively fuse spatial and temporal features in a more integrated manner. Our approach allows temporal features to dynamically attend to relevant spatial regions, enhancing the model's ability to detect fine-grained, time-evolving artifacts such as flickering eyes or warped lips. This design enables more precise localization and deeper contextual understanding, leading to improved performance across diverse and challenging scenarios. We evaluate the performance of our model using the FaceForensics++, Celeb-DF, and DeepfakeDetection datasets in both intra- and cross-dataset settings to affirm the superiority of our approach. Our model achieves strong performance with an AUC of 99.49 percent and an accuracy of 97.57 percent in intra-dataset evaluations. In cross-dataset testing, it demonstrates impressive generalization by achieving a 93.31 percent AUC on the unseen DeepfakeDetection dataset. These results highlight the effectiveness of cross-attention-based feature fusion in enhancing the robustness of deepfake video detection.

Key findings

The CAST model achieved state-of-the-art performance in intra-dataset evaluations on FaceForensics++ (99.49% AUC, 97.57% accuracy) and demonstrated strong generalization in cross-dataset testing, notably achieving 93.31% AUC on the unseen DeepfakeDetection dataset. Ablation studies confirmed the critical role of the cross-attention mechanism in enhancing generalization, particularly when temporal features serve as queries to guide spatial feature refinement.

Approach

The CAST model utilizes a CNN backbone to extract spatial features from individual video frames and a Transformer encoder to capture temporal dependencies across the sequence. A multi-head cross-attention module is then employed to fuse these spatial embeddings and Transformer-encoded temporal tokens, enabling temporal features to dynamically attend to relevant spatial regions. This integrated fusion enhances the detection of fine-grained, time-evolving manipulation artifacts.

Datasets

FaceForensics++, DeepfakeDetection (DFD), Celeb-DF (v2)

Model(s)

UNKNOWN

Author countries

India

← Previous