Search papers, labs, and topics across Lattice.
This paper investigates the use of adaptive perturbation selection in contrastive audio decoding to mitigate hallucinations in large audio-language models (LALMs). By evaluating a diverse library of structured audio transformations, the authors demonstrate that task-specific perturbations can significantly enhance model performance, achieving a notable accuracy increase from 74.7% to 81.4% on temporal order tasks. Furthermore, a lightweight perturbation selector trained on model hidden states leads to an additional 4.3% improvement on the existence task, showcasing the importance of tailored perturbations in audio decoding.
Task-specific audio perturbations can boost model accuracy by over 6% and reduce hallucinations in large audio-language models.
Large audio-language models (LALMs) frequently hallucinate by overriding acoustic evidence with language priors. While contrastive decoding (CD) offers training-free mitigation, existing methods rely on blunt perturbations like masking or noise, leaving structured audio transformations unexplored. We explore this design space by evaluating a diverse library of targeted audio perturbations and adaptively selecting the optimal negative branch for each task and example. First, we improve upon earlier prompt engineering by showing that a simple binary yes/no constraint reduces the model's tendency to falsely confirm absent audio features. Second, evaluating our library across temporal, spectral, frequency, and amplitude domains reveals that optimal transformations are highly task-dependent; for instance, reversing the audio array disrupts temporal coherence, raising accuracy on the temporal order task from 74.7% to 81.4%. Finally, we trained a light-weight perturbation selector on model hidden states to dynamically route negative branches, yielding an additional +4.3% gain on the existence task.
Google Research