Using Deepfake Technologies for Word Emphasis Detection

View on arXiv ← Back to list

Authors: Eran Kaufman, Lee-Ad Gottlieb

Published: 2023-05-12 22:50:53+00:00

AI Summary

This paper proposes a novel approach for automated emphasis detection in spoken language using deepfake technology. By generating an 'emphasis-devoid' version of a spoken sentence using a speaker's voice sample and comparing it to the original, the system identifies emphasized words.

Abstract

In this work, we consider the task of automated emphasis detection for spoken language. This problem is challenging in that emphasis is affected by the particularities of speech of the subject, for example the subject accent, dialect or voice. To address this task, we propose to utilize deep fake technology to produce an emphasis devoid speech for this speaker. This requires extracting the text of the spoken voice, and then using a voice sample from the same speaker to produce emphasis devoid speech for this task. By comparing the generated speech with the spoken voice, we are able to isolate patterns of emphasis which are relatively easy to detect.

Key findings

The proposed system achieved an accuracy, precision, recall, and F1-score of 92%, 89.14%, 89.33%, and 89.23%, respectively, demonstrating the effectiveness of the deepfake approach for emphasis detection. The system addresses the challenge of speaker-specific emphasis variations.

Approach

The approach uses a deepfake-based method. It generates an emphasis-devoid version of a spoken sentence using a speaker's voice sample, a speech-to-text (STT) module, and a text-to-speech (TTS) module. Emphasis is detected by comparing the generated speech with the original via FFT cross-correlation, identifying pitch shifts or skew.

Datasets

VCTK dataset (for training the TTS model), a custom dataset of 100 voice samples with varying sentence emphasis (for testing).

Model(s)

SV2TTS architecture (speaker encoder, sequence-to-sequence synthesizer, autoregressive WaveNet vocoder), Speech-to-Text (STT) and Fast Fourier Transform (FFT).

Author countries

Israel

← Previous