Search papers, labs, and topics across Lattice.
This paper addresses the challenge of speaker verification performance degradation when processing whispered speech, which differs acoustically from normal speech. They propose an encoder-decoder model, fine-tuned from a speaker verification backbone and jointly optimized with cosine similarity classification and triplet loss, to generate more robust representations against whispered speech. The proposed system achieves a 22.26% relative improvement over the baseline in normal vs. whispered speech trials and a 15% relative improvement over ReDimNet-B2 in whispered vs. whispered trials.
Speaker verification systems can be made significantly more robust to whispered speech by using a simple encoder-decoder architecture and a joint training objective.
Speaker verification is a task of confirming an individual's identity through the analysis of their voice. Whispered speech differs from phonated speech in acoustic characteristics, which degrades the performance of speaker verification systems in real-life scenarios, including avoiding fully phonated speech to protect privacy, disrupt others, or when the lack of full vocalization is dictated by a disease. In this paper we propose a model with a training recipe to obtain more robust representations against whispered speech hindrances. The proposed system employs an encoder--decoder structure built atop a fine-tuned speaker verification backbone, optimized jointly using cosine similarity--based classification and triplet loss. We gain relative improvement of 22.26\% compared to the baseline (baseline 6.77\% vs ours 5.27\%) in normal vs whispered speech trials, achieving AUC of 98.16\%. In tests comparing whispered to whispered, our model attains an EER of 1.88\% with AUC equal to 99.73\%, which represents a 15\% relative enhancement over the prior leading ReDimNet-B2. We also offer a summary of the most popular and state-of-the-art speaker verification models in terms of their performance with whispered speech. Additionally, we evaluate how these models perform under noisy audios, obtaining that generally the same relative level of noise degrades the performance of speaker verification more significantly on whispered speech than on normal speech.
