Search papers, labs, and topics across Lattice.
This paper analyzes the reasoning dynamics of 18 VLMs by tracking confidence, measuring corrective behavior, and evaluating intermediate reasoning steps within Chain-of-Thought (CoT). The key finding is that VLMs exhibit "answer inertia," where initial predictions are reinforced, and that while reasoning-trained models show better corrective behavior, they are still susceptible to misleading textual cues even with sufficient visual evidence. Furthermore, the detectability of this textual influence in CoT varies across models, with longer, fluent CoTs in reasoning-trained models obscuring modality reliance.
VLMs are more easily swayed by misleading text than you think, and their impressive reasoning chains can mask, rather than reveal, this over-reliance on language.
Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear. We analyze reasoning dynamics in 18 VLMs covering instruction-tuned and reasoning-trained models from two different model families. We track confidence over Chain-of-Thought (CoT), measure the corrective effect of reasoning, and evaluate the contribution of intermediate reasoning steps. We find that models are prone to answer inertia, in which early commitments to a prediction are reinforced, rather than revised during reasoning steps. While reasoning-trained models show stronger corrective behavior, their gains depend on modality conditions, from text-dominant to vision-only settings. Using controlled interventions with misleading textual cues, we show that models are consistently influenced by these cues even when visual evidence is sufficient, and assess whether this influence is recoverable from CoT. Although this influence can appear in the CoT, its detectability varies across models and depends on what is being monitored. Reasoning-trained models are more likely to explicitly refer to the cues, but their longer and fluent CoTs can still appear visually grounded while actually following textual cues, obscuring modality reliance. In contrast, instruction-tuned models refer to the cues less explicitly, but their shorter traces reveal inconsistencies with the visual input. Taken together, these findings indicate that CoT provides only a partial view of how different modalities drive VLM decisions, with important implications for the transparency and safety of multimodal systems.
