Visualizing Classifier Adjacency Relations: A Case Study in Speaker Verification and Voice Anti-Spoofing

View on arXiv ← Back to list

Authors: Tomi Kinnunen, Andreas Nautsch, Md Sahidullah, Nicholas Evans, Xin Wang, Massimiliano Todisco, Héctor Delgado, Junichi Yamagishi, Kong Aik Lee

Published: 2021-06-11 13:03:33+00:00

AI Summary

The paper proposes a novel method for visualizing the relationships between different binary classifiers using a 2D representation derived from rank correlations of their detection scores. This visualization aids in understanding classifier behavior, similarity, and complementarity, particularly useful in scenarios with limited classifier metadata, such as public machine learning challenges.

Abstract

Whether it be for results summarization, or the analysis of classifier fusion, some means to compare different classifiers can often provide illuminating insight into their behaviour, (dis)similarity or complementarity. We propose a simple method to derive 2D representation from detection scores produced by an arbitrary set of binary classifiers in response to a common dataset. Based upon rank correlations, our method facilitates a visual comparison of classifiers with arbitrary scores and with close relation to receiver operating characteristic (ROC) and detection error trade-off (DET) analyses. While the approach is fully versatile and can be applied to any detection task, we demonstrate the method using scores produced by automatic speaker verification and voice anti-spoofing systems. The former are produced by a Gaussian mixture model system trained with VoxCeleb data whereas the latter stem from submissions to the ASVspoof 2019 challenge.

Key findings

Visualizations reveal trends in classifier similarity based on known metadata (ASV) and provide insights into the diversity of systems in an uncontrolled setting (anti-spoofing). The approach offers a new perspective on classifier relationships, complementing traditional DET and ROC analyses.

Approach

The method uses Kendall's τ to calculate rank correlations between classifiers' detection scores on a common dataset. These correlations are then transformed into distances and visualized in 2D space using multidimensional scaling (MDS), allowing for visual comparison of classifiers.

Datasets

VoxCeleb, LibriSpeech, ASVspoof 2019 challenge dataset

Model(s)

Gaussian Mixture Model (GMM)-Universal Background Model (UBM)

Author countries

Finland, France, France, France, Japan, Spain, Singapore

← Previous