Search papers, labs, and topics across Lattice.
This paper presents the Whisper-Aware LLM, a novel framework designed to enhance automatic speech recognition (ASR) for whispered speech by addressing the inherent uncertainty in acoustic signals. By employing self-supervised tasks, the model learns to quantify and react to this uncertainty, leading to a significant reduction in both character error rates (CER) and hallucination rates during transcription. The results demonstrate a 17% relative CER improvement on the AISHELL6-Whisper dataset and a dramatic decrease in hallucination from over 25% to 4.5%, showcasing the model's robustness in challenging audio conditions.
Whispered speech recognition just got a major upgrade, slashing hallucination rates from over 25% to just 4.5% with a new self-supervised uncertainty learning approach.
The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whispered speech or hallucinatory transcription of noise. This paper introduces the Whisper-Aware LLM, a framework that teaches an Audio-LLM to perceive and react to this uncertainty. Our model develops an intrinsic self-awareness by learning to quantify the physical deficiencies of acoustic signals through targeted self-supervised tasks. This learned uncertainty is then operationalized via a novel Confidence-Fused Decoding mechanism, which provides both high-level instructions and frame-level attention modulation to the LLM decoder. Our experiments confirm the effectiveness of this approach. The model sets a new state-of-the-art on whispered speech with a 17% relative CER reduction on AISHELL6-Whisper. At the same time, it directly addresses the reliability trade-off, with hallucination rates dropping from over 25% to 4.5%.
