Search papers, labs, and topics across Lattice.
This paper introduces Likelihood-Constrained Acoustic Reranking (LCAR), a novel decoding method aimed at mitigating hallucinations in LLM-based automatic speech recognition (ASR) systems without requiring additional training. By leveraging a likelihood margin and acoustic compatibility scores, LCAR effectively retains and reranks tokens to enhance acoustic grounding while preserving the support of the base model. The evaluation shows that LCAR significantly reduces hallucination failures by 38.8–57.1% while maintaining word and character error rates on standard test sets, demonstrating its efficacy in challenging conditions.
LCAR slashes hallucination failures by over 50% in LLM-based ASR systems without any additional training or external models.
Large language model (LLM)-based automatic speech recognition (ASR) systems achieve strong performance on conventional speech data by leveraging powerful linguistic priors and multilingual capabilities. However, under challenging conditions, these priors can override acoustic evidence, resulting in unintended translation, instruction execution, repetition, or catastrophic deletion. We propose Likelihood-Constrained Acoustic Reranking (LCAR), a training-free decoding method that improves acoustic grounding while preserving support from the base model. At each decoding step, LCAR first retains tokens whose base-model likelihood falls within a margin of the greedy token, then reranks them using an acoustic compatibility score computed from attention-pooled audio embeddings and the existing LM head. By restricting acoustic intervention to plausible, model-supported alternatives, LCAR requires no additional training, external detector, reference transcript, or auxiliary model at inference. We evaluate LCAR on four LLM-based ASR systems using human-audited TTS and open-source speech challenge suites. At $δ=0.60$, LCAR removes 38.8--57.1\% of detector-identified hallucination failures while largely maintaining WER/CER on standard open-source test sets.
