Self-Attention and Hybrid Features for Replay and Deep-Fake Audio Detection

View on arXiv ← Back to list

Authors: Lian Huang, Chi-Man Pun

Published: 2024-01-11 01:41:16+00:00

AI Summary

This paper proposes a novel framework for replay and deepfake audio detection using hybrid features and a self-attention mechanism. The approach combines deep learning features and Mel-spectrogram features, leveraging self-attention to focus on essential elements for improved discrimination. This results in significantly lower Equal Error Rates (EERs) compared to baseline systems on the ASVspoof 2021 dataset.

Abstract

Due to the successful application of deep learning, audio spoofing detection has made significant progress. Spoofed audio with speech synthesis or voice conversion can be well detected by many countermeasures. However, an automatic speaker verification system is still vulnerable to spoofing attacks such as replay or Deep-Fake audio. Deep-Fake audio means that the spoofed utterances are generated using text-to-speech (TTS) and voice conversion (VC) algorithms. Here, we propose a novel framework based on hybrid features with the self-attention mechanism. It is expected that hybrid features can be used to get more discrimination capacity. Firstly, instead of only one type of conventional feature, deep learning features and Mel-spectrogram features will be extracted by two parallel paths: convolution neural networks and a short-time Fourier transform (STFT) followed by Mel-frequency. Secondly, features will be concatenated by a max-pooling layer. Thirdly, there is a Self-attention mechanism for focusing on essential elements. Finally, ResNet and a linear layer are built to get the results. Experimental results reveal that the hybrid features, compared with conventional features, can cover more details of an utterance. We achieve the best Equal Error Rate (EER) of 9.67% in the physical access (PA) scenario and 8.94% in the Deep fake task on the ASVspoof 2021 dataset. Compared with the best baseline system, the proposed approach improves by 74.60% and 60.05%, respectively.

Key findings

The proposed method achieved the best EER of 9.67% in the physical access (PA) scenario and 8.94% in the deepfake task on the ASVspoof 2021 dataset. This represents a substantial improvement (74.60% and 60.05%, respectively) over the best baseline systems. The self-attention mechanism played a crucial role in achieving these results.

Approach

The authors propose a two-path framework. One path extracts deep features using CNNs, and the other extracts Mel-spectrogram features using STFT. These features are concatenated, processed by a self-attention mechanism to focus on important elements, and then fed into a ResNet for classification.

Datasets

ASVspoof 2019 and ASVspoof 2021 datasets

Model(s)

Convolutional Neural Networks (CNNs), ResNet, Self-attention mechanism

Author countries

China, China

← Previous