Search papers, labs, and topics across Lattice.
This study explores the use of zero-shot voice cloning as a data augmentation technique for automatic speech recognition (ASR) systems tailored to dysarthric speech, addressing the challenges of data scarcity and inter-speaker variability. By fine-tuning the Whisper-medium model on a combination of cloned, real, and hybrid datasets from the TORGO dataset, the authors demonstrate that their approach yields competitive word error rates (WER), particularly for moderate to severe dysarthric speakers. The findings reveal that zero-shot voice cloning can effectively supplement training data, achieving superior performance in cross-corpus evaluations compared to traditional methods.
Zero-shot voice cloning can significantly enhance ASR performance for dysarthric speech, outperforming traditional data collection methods with minimal burden.
Automatic speech recognition remains unreliable for dysarthric speech due to data scarcity and high inter-speaker variability. While synthetic data can address these gaps, traditional methods often require extensive speaker-specific data, reintroducing the collection bottleneck. We investigate zero-shot voice cloning as a low-burden augmentation strategy, using Higgs Audio V2 to clone speakers in the TORGO dataset. We fine-tune (FT) Whisper-medium on cloned, real, and hybrid data and evaluate on held-out real speech. Compared to the zero-shot (31.62%), Clone FT achieved a competitive 26.00% WER, nearly matching the 24.44% and 25.12% seen with Real and Hybrid FT, respectively. Notably, Clone and Hybrid FT outperform Real FT for moderate-severe speakers. Clone FT achieves the best results (11.45% relative) in cross-corpus evaluation on the SAP-1102. These results suggest that zero-shot cloning provides scalable training data that circumvents the costly data collection bottleneck.
