Search papers, labs, and topics across Lattice.
This study investigates the text dominance bias in Audio Large Language Models (Audio LLMs), where models prioritize textual inputs over acoustic evidence, leading to hallucinations. By conducting a mechanistic analysis of internal representations, the authors identify that while text and audio utilize distinct pathways, they converge in late layers, with the text pathway actively suppressing audio information. The introduction of back-patching, a novel intervention that enhances audio representations, effectively reduces text dominance and promotes better multimodal alignment in conflicting scenarios.
Text dominance in Audio LLMs can be mitigated through a novel back-patching technique that enhances audio representations, challenging the status quo of multimodal processing.
While Audio Large Language Models (Audio LLMs) excel at multimodal understanding, they suffer from text dominance, a bias where models blindly favor text over acoustic evidence, causing hallucinations. However, the internal mechanisms underlying how these models behave when audio and textual inputs contradict each other remain unexplored. In this work, we present the first mechanistic analysis of this phenomenon by tracing the propagation of internal representations across layers. Our investigation reveals three key findings: (i) text dominance is systematically and empirically across models; (ii) while text and audio rely on functionally distinct pathways, they ultimately converge into a shared semantic space in late layers; and (iii) the text pathway does not erase audio information, but rather actively suppresses intact audio representations. Building on these insights, we leverage back-patching, a training-free intervention that routes late-layer audio activations back into earlier layers. This amplifies the audio representations, enabling them to overcome textual suppression. Our evaluation shows that back-patching consistently reduces text dominance, paving the way for mechanistic multimodal alignment under conflict.
