What You Train Is What You Get: Gender Bias, Training Composition, and Post-Hoc Mitigation in Audio Deepfake Detection

Authors: Aishwarya R. Fursule, Vamshi Nallaguntla, Shruti Kshirsagar, Anderson R. Avila

Published: 2026-07-10 18:29:24+00:00

Comment: This manuscript is a preprint and is currently under review for the Special Section on Trustworthy and Reliable AI in IEEE Transactions on Reliability

AI Summary

This research investigates gender bias in audio deepfake detection using the ASVspoof5 dataset, demonstrating that training data composition strongly predicts bias direction and that the underrepresented gender performs worse at test time. It finds that WavLM-Base+ features exhibit significantly larger gender performance gaps than LogSpectrogram, and critically, post-hoc threshold calibration methods are ineffective at mitigating these distribution-level disparities, suggesting that fairness must be addressed during training.

Abstract

Audio deepfake detection models determine whether speech is genuine or artificially generated, but high overall accuracy can mask substantial performance disparities across demographic groups. In this work, we investigate gender bias in audio deepfake detection using the ASVspoof5 dataset. We use ASVspoof5 under a controlled custom split designed to isolate gender-composition effects. We train attack-specific models on nine training sets with different gender compositions, ranging from female-only to male-only. We use a ResNet18 classifier with LogSpectrogram and WavLM-Base+ features, and we evaluated six post-hoc threshold calibration methods. Experimental results show that training data composition strongly predicts bias direction, with the underrepresented gender performing worse at test time. WavLM-Base+ features are shown to produce gender performance gaps 3.0 to 4.3 times larger than LogSpectrogram under identical training conditions, and balanced training is found to reduce LogSpectrogram bias but leave WavLM bias largely intact. Moreover, all six calibration strategies, including Oracle calibration with full test-set label access, leave the Equal Error Rate gap unchanged at 1.317 pp, confirming that threshold adjustment cannot correct underlying score distribution disparities. Overall, these findings suggest that gender fairness in audio deepfake detection must be addressed at training time, as post-hoc methods can only partially mitigate the resulting disparities

Key findings

Training data composition strongly predicts bias direction, with the underrepresented gender performing worse. WavLM-Base+ features produce gender performance gaps 3.0 to 4.3 times larger than LogSpectrogram, and balanced training disproportionately benefits LogSpectrogram. Moreover, all six post-hoc threshold calibration strategies, including Oracle calibration, failed to reduce the Equal Error Rate gap, indicating that fairness must be addressed at the training stage due to underlying score distribution disparities.

Approach

The study trains attack-specific ResNet18 models using LogSpectrogram and WavLM-Base+ features on ASVspoof5 with custom gender-composition splits, ranging from female-only to male-only training sets. It then evaluates the resulting gender bias using multiple fairness metrics and assesses the effectiveness of six post-hoc threshold calibration strategies.

Datasets

ASVspoof5

Model(s)

ResNet18, WavLM-Base+

Author countries

USA, Canada

← Previous