Search papers, labs, and topics across Lattice.
This paper introduces See2Think, a comprehensive evaluation framework designed to assess the reliance of multimodal large language models on intermediate visual states during reasoning. By employing See2ThinkBench, which includes 1,200 visually dependent problems across diverse task categories, and Visual Action-of-Thought (VAoT) to capture the reasoning process, the authors reveal that visual reasoning performance varies significantly based on the model and environment. Notably, while models often select relevant visual operations, challenges in faithful rendering and the impact of corrupted feedback highlight critical limitations in their visual reasoning capabilities, leading to substantial drops in accuracy under certain conditions.
Visual reasoning in multimodal models is highly variable, with accuracy plummeting over 10 points when feedback is corrupted, revealing hidden dependencies on visual states.
Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states. Existing benchmarks are limited both by task collections with narrow coverage or partially text-solvable samples and by evaluations that emphasize final answers without diagnosing how intermediate visual states are generated, rendered, and used. We introduce See2Think, a unified evaluation framework comprising See2ThinkBench and Visual Action-of-Thought (VAoT). See2ThinkBench contains 1,200 open-ended, visually dependent problems across 12 task categories spanning 2D structured, 3D scene, and real-world reasoning. VAoT records textual thoughts, visual actions, rendered states, and subsequent reasoning under four controlled inference settings. Evaluating representative proprietary and open-source multimodal models, we find that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks. Process analysis further shows that models usually select relevant visual operations, while faithful rendering remains the clearest bottleneck and high feedback uptake does not necessarily translate into accuracy gains. Under task-relevant corrupted feedback, models exhibit behavioral dependence on visual states, with accuracy dropping by over 10 percentage points in controlled interventions.
