Search papers, labs, and topics across Lattice.
This study develops a severity-aware automatic speech recognition (ASR) framework specifically for individuals with cleft lip and palate (CLP), addressing the challenges posed by limited pathological speech data and variability in speech characteristics. By fine-tuning the Whisper-small model with diverse CLP speech samples across varying severity levels, the researchers achieved a significant reduction in word error rate (WER) from 62.46% to 22.72%, while also enhancing performance consistency across different severity groups. The deployment on an NVIDIA Jetson platform enables real-time, edge-based ASR, making voice interaction more accessible for individuals with CLP in scenarios where cloud services are unavailable.
Incorporating severity diversity in ASR training can slash error rates by over 60%, leveling the playing field for individuals with cleft lip and palate.
Automatic speech recognition (ASR) remains challenging for individuals with cleft lip and palate (CLP) because of limited pathological speech data and large variations in speech characteristics across speakers and severity levels. These recognition difficulties can reduce the accessibility of voice-based human-computer interaction, particularly when cloud-based ASR services are unavailable or unreliable. This work investigates a severity-aware and edge-deployable ASR framework for improving recognition of CLP speech using Whisper-small. The model was fine-tuned using different combinations of normal and CLP speech representing mild, moderate, and severe conditions, together with a CLP-only training configuration, to examine how the inclusion of different severity levels influences recognition performance and fairness across speakers. The pretrained model produced pooled word error rate (WER) and phoneme error rate (PER) values of 62.46% and 52.72%, respectively. Severity-aware fine-tuning substantially improved performance, reducing the best pooled WER to 22.72% and the best pooled PER to 18.44%. Training with a broader representation of CLP severity levels also provided the best overall balance between recognition accuracy and performance consistency across severity groups. Deployment on an NVIDIA Jetson platform demonstrated real-time inference for all fine-tuned models, with real-time factors of 0.167-0.171 and peak GPU memory usage of approximately 566 MB. The results demonstrate that incorporating severity diversity during ASR adaptation can substantially improve recognition of CLP speech while reducing performance disparities across severity groups. The proposed approach further enables low-latency, Internet-independent speech interaction on edge devices, supporting more accessible and inclusive voice-based human-computer interaction for individuals with CLP.
