TranssionADD: A multi-frame reinforcement based sequence tagging model for audio deepfake detection

View on arXiv ← Back to list

Authors: Jie Liu, Zhiba Su, Hui Huang, Caiyan Wan, Quanxiu Wang, Jiangli Hong, Benlai Tang, Fengjie Zhu

Published: 2023-06-27 05:18:25+00:00

AI Summary

This paper proposes TranssionADD, a multi-frame reinforcement-based sequence tagging model for audio deepfake detection. It improves model robustness and handles outliers by using a multi-frame detection module and an isolated-frame penalty loss, achieving 2nd place in Track 2 of the ADD 2023 challenge.

Abstract

Thanks to recent advancements in end-to-end speech modeling technology, it has become increasingly feasible to imitate and clone a user`s voice. This leads to a significant challenge in differentiating between authentic and fabricated audio segments. To address the issue of user voice abuse and misuse, the second Audio Deepfake Detection Challenge (ADD 2023) aims to detect and analyze deepfake speech utterances. Specifically, Track 2, named the Manipulation Region Location (RL), aims to pinpoint the location of manipulated regions in audio, which can be present in both real and generated audio segments. We propose our novel TranssionADD system as a solution to the challenging problem of model robustness and audio segment outliers in the trace competition. Our system provides three unique contributions: 1) we adapt sequence tagging task for audio deepfake detection; 2) we improve model generalization by various data augmentation techniques; 3) we incorporate multi-frame detection (MFD) module to overcome limited representation provided by a single frame and use isolated-frame penalty (IFP) loss to handle outliers in segments. Our best submission achieved 2nd place in Track 2, demonstrating the effectiveness and robustness of our proposed system.

Key findings

TranssionADD significantly outperforms baseline models in terms of both detection accuracy and the reduction of isolated segments (outliers). Ablation studies confirm the contribution of each component of the proposed system, particularly the MFD module and IFP loss.

Approach

TranssionADD adapts a sequence tagging task for audio deepfake detection, incorporating a multi-frame detection (MFD) module to leverage contextual information and an isolated-frame penalty (IFP) loss to mitigate the impact of outliers. Data augmentation techniques are employed to enhance model generalization.

Datasets

A publicly available Mandarin corpus provided by the organizers of ADD 2023.

Model(s)

RCNN-BLSTM (Residual Convolutional Neural Network and Bidirectional Long Short-Term Memory) with a multi-frame detection module.

Author countries

China

← Previous