Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces HALAS, the first human-annotated dataset capturing naturally occurring hallucinations in state-of-the-art Automatic Speech Recognition (ASR) systems, specifically from unprocessed earnings call recordings. The dataset features span-level labels that facilitate in-depth analysis of hallucination patterns and severity, revealing significant vocabulary overlap across models and the prevalence of hallucinations even in nearly accurate transcriptions. The benchmark established by HALAS demonstrates that existing character and semantic-level metrics for hallucination detection achieve an 81% ROC-AUC, significantly outperforming state-of-the-art methods, which only reach an F1 score of 53.1%.
HALAS reveals that ASR systems hallucinate even with low Word Error Rates, challenging the effectiveness of current detection methods.
End-to-end Automatic Speech Recognition (ASR) systems hallucinate on natural speech, yet existing mitigation methods are typically evaluated on non-speech or artificially corrupted audio. We introduce HALAS, the first human-annotated dataset of naturally occurring hallucinations from seven state-of-the-art ASR models on real unprocessed earnings call recordings. HALAS provides span-level labels, enabling analysis of hallucination patterns and their severity. Our analysis reveals strong cross-model vocabulary overlap and confirms that hallucinations also occur for almost correctly transcribed speech (characterized by a low Word Error Rate). The proposed benchmark with HALAS shows that the character and semantic-level metrics used as a proxy for hallucination detection reach 81% ROC-AUC, while state-of-the-art detection methods achieve an F1 score of only 53.1%. As such, HALAS establishes the first rigorous non-artificial benchmark for the detection and mitigation of ASR hallucinations.