Search papers, labs, and topics across Lattice.
This paper introduces a reinforcement learning framework, VRRL, designed to enhance visually grounded self-reflection in vision-language models by addressing their shortcomings in attending to visual inputs during the reflection process. By employing techniques such as random masking of trajectory prefixes and utilizing an experience replay buffer, the model learns to recover from incorrect predictions and adapt to diverse failure states. The results demonstrate that VRRL significantly improves out-of-distribution accuracy on visual grounding tasks compared to standard RL and reflection-oriented fine-tuning methods, showcasing its effectiveness in real-world applications.
Vision-language models can now achieve remarkable out-of-distribution accuracy by effectively leveraging self-reflection to correct errors in multimodal reasoning.
Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images. To address this issue, we propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection. First, we randomly mask trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions rather than making early mistakes. Second, we introduce buffered roll-ins from an experience replay buffer to expose the model to diverse failure states that it must learn to correct. We evaluate our approach on visual grounding tasks involving tables and charts, as well as spatial navigation benchmarks. While off-the-shelf and conventionally fine-tuned models degrade substantially under distribution shift, our method substantially improves average out-of-distribution accuracy over standard RL and reflection-oriented fine-tuning baselines by using self-reflection effectively.
