Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates hallucination detection in the Whisper ASR model using three distinct paradigms: text-based classifiers, LLM-based methods, and probing internal decoder states. While text classifiers show high recall, they falter without reference transcripts, and LLM-based approaches improve precision but lag behind text methods. The most effective detection leverages internal decoder representations, revealing that hallucination characteristics are embedded in the model's intermediate layers, with a late-fusion meta-classifier achieving the highest overall performance.
Internal decoder states hold the key to detecting ASR hallucinations more effectively than traditional text-based methods.
Hallucinations of ASR models - fluent transcriptions with no basis in audio - degrade system performance and pose risks in downstream applications. Robust detection of such errors remains a challenge. This paper studies Whisper large v3 hallucination detection on real-speech human-annotated data across three paradigms: text-based, LLM-based, and internal decoder state probing. Text classifiers utilizing metrics for text evaluation achieve high recall but degrade without reference transcripts. LLM-based detection improves precision with domain-specific prompt conditioning, yet remains less competitive than the lightweight text-based methods. Probing Whisper's decoder representations, without a ground-truth reference, yields the strongest performance, revealing that hallucination traits are encoded across intermediate decoding layers. A late-fusion meta-classifier combining text and internal-state outputs achieves the best overall detection performance.