Search papers, labs, and topics across Lattice.
This paper introduces Head-Wise Representation Alignment (HeRA), a novel method for enhancing Multimodal Large Language Models (MLLMs) by aligning individual attention heads with an external vision encoder. By focusing on the topological structure of representations and employing a contrastive objective based on the Mutual K-Nearest Neighbor (MKNN) alignment metric, HeRA selectively aligns the least aligned heads during training, leading to significant performance improvements on vision-centric tasks. The results reveal that this targeted alignment not only boosts task performance but also reduces visual hallucinations, demonstrating the efficacy of fine-grained representation alignment in MLLMs.
Aligning the least aligned attention heads in MLLMs can yield the most significant performance gains, challenging conventional alignment strategies.
Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.
