Search papers, labs, and topics across Lattice.
This study addresses the challenge of reliable speaker verification (SV) in expressive text-to-speech (TTS) and voice conversion (VC) systems, particularly focusing on non-verbal vocalizations (NVVs). By integrating frozen Data2Vec features with an ECAPA-TDNN architecture and a Mixture of Experts (MoE) module, the authors effectively mitigate catastrophic forgetting while enhancing identity consistency across verbal and non-verbal segments. The proposed framework achieves a significant reduction in speech-NVV equal error rate (EER) from 38.93% to 22.66% and improves speech EER from 13.17% to 9.24%, demonstrating its effectiveness in maintaining performance across domains.
Bridging the gap between verbal and non-verbal vocalizations, this approach slashes speaker verification errors by over 40% while preserving speech accuracy.
As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.
