Why Do You Say It Like That? A Phoneme-Level Framework for Explainable Speech Deepfake Detection

Authors: Anna Taylor, Michele Panariello, Massimiliano Todisco, Chiara Galdi, Nicholas Evans, Driss Matrouf

Published: 2026-07-09 15:23:45+00:00

AI Summary

This paper introduces a phoneme-level analysis framework for explainable speech deepfake detection. It leverages Grad-CAM with speech recognition to generate saliency maps aligned with phonemes and pauses, providing linguistic interpretations for model predictions. The framework reveals statistically significant attack- and speaker-dependent phonetic cues associated with spoofed speech.

Abstract

As the accuracy of speech deepfake detection improves with the use of self-supervised representations such as wav2vec 2.0 and HuBERT, understanding why the speech is classified as bona fide or deepfake remains an open challenge. In pursuit of more trustworthy and interpretable artificial intelligence, we introduce a phoneme-level analysis framework that connects model predictions to measurable phonetic units. Our post-hoc explainability method is generally applicable to a variety of speech deepfake detection systems based on convolutional neural networks since it leverages Gradient-weighted Class Activation Mapping in conjunction with speech recognition to generate saliency maps aligned with phonemes and pauses. This pipeline reveals statistically significant attack- and speaker-dependent phonetic cues associated with spoofed speech in terms that humans can understand. Experiments using ASVspoof 5 show comparable detection performance to similar architectures while providing linguistic interpretations across speakers and spoofing conditions.

Key findings

The framework demonstrates that modern deepfake detectors exploit a complex combination of phonetic, attack-dependent, and speaker-dependent information, with vowels, fricatives, approximants, and non-speech regions being particularly informative. Significant differences were observed in phoneme importance and temporal localization of activations across various spoofing attacks and speakers. The model notably relies on acoustic transitions for natural speech detection and steady-state phonetic realizations for spoofed speech.

Approach

The approach combines a self-supervised learning (SSL) based front-end (WavLM), a temporal Convolutional Neural Network (CNN) classifier, and a post-hoc explainability module based on Grad-CAM. By aligning temporal saliency maps with phoneme boundaries obtained through automatic speech recognition and forced alignment, the framework analyzes model activations in linguistically meaningful units.

Datasets

ASVspoof 5

Model(s)

WavLM Base+ (as SSL front-end), temporal CNN (as back-end classifier)

Author countries

France

← Previous