Search papers, labs, and topics across Lattice.
The paper introduces a training-free, iterative framework called "See It, Say It, Sorted" to improve visually-grounded multimodal reasoning in LVLMs by mitigating visual hallucination. The method supervises each reasoning step at test time using a textual visual-evidence pool, ensuring that each generated token is justified by visual cues. A visual decider module dynamically extracts additional relevant evidence from the image when existing evidence is insufficient, improving reasoning accuracy and reducing hallucination rates without requiring model training.
LVLMs can get a 16.5%-29.5% boost on visual reasoning benchmarks just by iteratively checking their work against the image, no training needed.
Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.
