Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the problem of video-driven audio hallucination in Audio-Visual Language Models (AVLMs), where models prioritize visual cues over actual audio information. They introduce Audio-Contrastive Preference Optimization (ACPO), a dual-axis preference learning framework with output-contrastive and input-contrastive objectives to penalize visually-driven audio generation. Experiments show ACPO significantly improves audio grounding and reduces hallucination while maintaining overall multimodal performance.
AVLMs are routinely ignoring the audio and hallucinating sounds based on video, but a new training method can fix this.
While Audio-Visual Language Models (AVLMs) have achieved remarkable progress over recent years, their reliability is bottlenecked by cross-modal hallucination. A particularly pervasive manifestation is video-driven audio hallucination: models routinely exploit visual shortcuts to hallucinate expected sounds, discarding true auditory evidence. To counteract this deeply ingrained visual dominance, we propose Audio-Contrastive Preference Optimization (ACPO). This dual-axis preference learning framework introduces an output-contrastive objective to penalize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize generation invariant to the true auditory signal. Extensive experiments demonstrate that ACPO establishes highly faithful audio grounding and mitigates audio hallucination without compromising overarching multimodal capabilities.