Search papers, labs, and topics across Lattice.
This paper introduces Information-Regularized Attention (IRA), a stochastic attention mechanism designed to enhance visual representation learning in vision-language models (VLMs) by explicitly regulating the visual information integrated into transformer layers. The authors demonstrate that IRA mitigates issues such as object hallucination and catastrophic forgetting by translating uncertainty into local noise, resulting in smoother curvature trajectories and reduced attention-sink across all layers. The findings indicate that IRA not only stabilizes the model's performance but also fundamentally improves the representation learning process in generative architectures, paving the way for more reliable VLMs.
Stochastic attention isn't just a regularizer; it fundamentally transforms how visual information is learned in VLMs, leading to more stable and reliable models.
Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning. We claim these failures result from a lack of explicit control over visual representation learning during the standard next-token prediction objective. As a result, visual embeddings thus become passively optimized and prone to injecting redundant or spurious signals. To counter this, we introduce Information-Regularized Attention (IRA), a stochastic attention mechanism that explicitly regulates the amount of visual information injected into the hidden states of intermediate transformer layers. This local reparameterization translates uncertainty about visual representations into local noise that is independent across data points. Beyond evaluating model performance, we also quantify embedding properties, where IRA produces smoother curvature trajectories and suppresses attention-sink across all layers, indicating a more stable transformation of the visual signal. Our results suggest that stochastic attention is not merely a regularizer but a key contributor to representation learning in a generative architecture, offering a new direction for building more reliable VLMs.
Meta AI