Search papers, labs, and topics across Lattice.
This paper addresses the issue of visual under-conditioning in self-evolving large multimodal models (LMMs) by introducing VISE (Visual Invariance Self-Evolution), which enhances the model's attention to visual tokens through two complementary invariance-based rewards. By enforcing spatial and semantic consistency, VISE significantly improves performance on vision-language tasks, such as image captioning and visual question answering, without relying on external annotations or specialized roles. Experimental results show that VISE achieves substantial gains in CIDEr scores and reduces object hallucination across multiple benchmarks and model families.
Visual under-conditioning in LMMs can be overcome by directly regularizing visual attention, leading to remarkable improvements in multimodal understanding tasks.
Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision--language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18 benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE
