Search papers, labs, and topics across Lattice.
This paper introduces Re-Sonance, an innovative speech-driven Augmentative and Alternative Communication (AAC) system that leverages a three-stage architecture combining Whisper ASR, Qwen LLM, and CosyVoice TTS to facilitate real-time communication for individuals with dysarthria. The system addresses the limitations of existing AAC technologies by significantly enhancing speech intelligibility and naturalness, particularly in professional speaking contexts. Evaluation results indicate that Re-Sonance notably improves speech clarity and semantic coherence for users with mild to moderate dysarthria, highlighting the promise of LLM integration in AAC applications.
Real-time speech conversion for dysarthric individuals can now achieve unprecedented intelligibility and naturalness, transforming professional communication.
Individuals with dysarthria face significant challenges in professional speaking scenarios such as conferences, presentations, and meetings, where real-time communication is crucial. While existing Augmentative and Alternative Communication (AAC) systems provide basic support, they often fail to meet the demands of professional speaking environments due to high latency and unnatural speech patterns. This paper presents Re-Sonance, a novel LLM-enhanced speech-driven AAC system designed for real-time professional speaking scenarios. By integrating Whisper ASR, Qwen LLM, and CosyVoice TTS, Re-Sonance achieves improved speech intelligibility and naturalness while maintaining real-time performance. Both subjective and objective evaluations using a Mandarin dysarthric speech dataset demonstrate that our speech reconstruction approach significantly improved intelligibility while preserving semantic coherence for speakers with mild to moderate dysarthria. Although performance remains limited for severe dysarthria cases, our findings validate the potential of LLM-based methods for enhancing speech-driven AAC systems, paving the way for more effective and accessible communication technologies.
