Search papers, labs, and topics across Lattice.
This paper introduces Visual Evidence Pre-Alignment (VEPA), a novel intermediate training stage that optimizes visual evidence descriptions using a sufficiency-driven objective with Group Relative Policy Optimization (GRPO). By addressing the limitations of traditional caption-based pretraining, VEPA significantly improves the integration of visual inputs in multimodal large language models (MLLMs), leading to more consistent and accurate responses. Extensive experiments demonstrate that VEPA enhances performance on visually demanding tasks without relying on additional task-specific training, highlighting the importance of robust visual grounding.
Visual Evidence Pre-Alignment boosts MLLM performance by ensuring models effectively utilize fine-grained visual evidence, rather than just relying on coarse captions.
Multimodal large language models (MLLMs) integrate strong text reasoning with visual inputs, yet their responses can be inconsistent with the underlying images, indicating ineffective utilization of visual evidence during inference. The prevailing training paradigm relies on large-scale caption-based pretraining for general alignment, followed by supervised fine-tuning and reinforcement learning to enable instruction following and complex reasoning. However, such pretraining provides only weak visual grounding: short, coarse captions bias models toward salient objects while neglecting fine-grained visual evidence. In this paper, we introduce Visual Evidence Pre-Alignment (VEPA), an intermediate stage between pretraining and post-training that explores a novel sufficiency-driven objective with Group Relative Policy Optimization (GRPO) to optimize question-conditioned visual evidence descriptions. Extensive experiments across diverse benchmarks show that our VEPA consistently enhances performance on visually demanding evaluations and complements standard supervised post-training. Further analyses show that the income stems from strengthened, transferable visual grounding, rather than from additional task-specific training.
