Representation Loss Minimization with Randomized Selection Strategy for Efficient Environmental Fake Audio Detection

View on arXiv ← Back to list

Authors: Orchid Chetia Phukan, Girish, Mohd Mujtaba Akhtar, Swarup Ranjan Behera, Nitin Choudhury, Arun Balaji Buduru, Rajesh Sharma, S. R Mahadeva Prasanna

Published: 2024-09-24 05:46:52+00:00

Comment: Submitted to ICASSP 2025

AI Summary

This paper addresses the high computational demands of environmental audio deepfake detection (EADD) due to the high dimensionality of foundation model representations. The authors propose a randomized selection strategy, showing that randomly selecting 40-50% of representation values can preserve or improve performance compared to full representations and SOTA dimensionality reduction techniques. This method significantly reduces model parameters and inference time by almost half.

Abstract

The adaptation of foundation models has significantly advanced environmental audio deepfake detection (EADD), a rapidly growing area of research. These models are typically fine-tuned or utilized in their frozen states for downstream tasks. However, the dimensionality of their representations can substantially lead to a high parameter count of downstream models, leading to higher computational demands. So, a general way is to compress these representations by leveraging state-of-the-art (SOTA) unsupervised dimensionality reduction techniques (PCA, SVD, KPCA, GRP) for efficient EADD. However, with the application of such techniques, we observe a drop in performance. So in this paper, we show that representation vectors contain redundant information, and randomly selecting 40-50% of representation values and building downstream models on it preserves or sometimes even improves performance. We show that such random selection preserves more performance than the SOTA dimensionality reduction techniques while reducing model parameters and inference time by almost over half.

Key findings

Randomly selecting 40-50% of foundation model representation values preserves or sometimes improves EADD performance while outperforming SOTA dimensionality reduction techniques. This strategy leads to a reduction of almost half in downstream model parameters and inference time. The behavior is consistent across various audio and multimodal foundation models, highlighting its broad applicability and efficiency gains.

Approach

The approach involves extracting high-dimensional representation vectors from various audio and multimodal foundation models. Instead of using traditional dimensionality reduction, the authors propose randomly selecting 40-50% of these representation values. Simple downstream models, specifically a Fully Connected Network (FCN) or a Convolutional Neural Network (CNN), are then trained on these reduced representations for binary classification (fake/non-fake).

Datasets

DCASE 2023 Challenge dataset

Model(s)

Foundation Models (for representations): Unispeech-SAT, WavLM, Wav2vec2, TRILLsson, LanguageBind, ImageBind, CLAP. Downstream Models: Fully Connected Network (FCN), 1D-CNN.

Author countries

India, Estonia

← Previous