DETECT-3B-Omni is Agnostic of Content and Demographics

Authors: Nicolas M. Müller, Aditya Tirumala Bukkapatnam, Dominik Schnieders, Zohaib Ahmed

Published: 2026-07-03 15:27:17+00:00

AI Summary

This study evaluates the semantic independence of Resemble AI's deepfake audio detector, DETECT-3B-Omni, ensuring it focuses on acoustic artifacts rather than content or speaker demographics. Using 10,240 diverse audio samples from 8 AI voice-cloning systems, the research demonstrates that the detector's accuracy is consistently high across varied content (benign vs. malicious) and speaker characteristics (gender, age, region). Equivalence testing with a tight 2 percentage-point margin and 99% confidence confirms the detector's agnosticism, making it suitable for GDPR-compliant deployments.

Abstract

A trustworthy and GDPR-compliant deepfake audio detector must base its decisions on acoustic artifacts, not on what is being said or who is speaking. We present a large-scale study of semantic independence for Resemble AI's detector, DETECT-3B-Omni. Using 10,240 audio samples from diverse US English speakers across 30 states, generated through 8 different AI voice-cloning systems, we test whether detection accuracy depends on spoken content (benign versus malicious), speaker gender, speaker age, or speaker region. Using equivalence testing, our results show that the accuracy difference between any two of these groups is at most 2 percentage points, at 99% confidence. The detector therefore identifies AI-generated audio with equivalent accuracy regardless of what the audio says or who the speaker is.

Key findings

The DETECT-3B-Omni detector achieved an overall accuracy of 98.3% on out-of-distribution audio. Equivalence testing demonstrated that detection accuracy differences across content types (benign vs. malicious) and speaker demographics (gender, age, region) were within ±2 percentage points at 99% confidence. This indicates that the detector relies on acoustic artifacts rather than semantic content or speaker identity, aligning with GDPR compliance requirements.

Approach

The researchers employed equivalence testing on a meticulously constructed dataset of 10,240 audio samples. These samples varied in content (benign vs. malicious) and speaker demographics (gender, age, region), and were generated by 8 different AI voice-cloning systems or recorded live. The DETECT-3B-Omni API classified each sample, and the accuracy differences between groups were analyzed using 99% confidence intervals with a stringent ±2 percentage-point equivalence margin to establish semantic independence.

Datasets

A custom dataset of 10,240 audio samples comprising 5,120 real and 5,120 fake recordings. The real audio was recorded by 8 diverse native US English speakers across 30 states, balanced by gender and aged 20-55. The fake audio was generated using 8 open-source text-to-speech voice-cloning models: Chatterbox, Chatterbox-Turbo, Qwen3-TTS 1.7B, Qwen3-TTS 0.6B, XTTS2, E2-TTS, F5-TTS, and MegaTTS3.

Model(s)

DETECT-3B-Omni (Resemble AI's deepfake audio detector)

Author countries

USA, Germany

← Previous