Search papers, labs, and topics across Lattice.
This study critically evaluates the effectiveness of the "thinking-with-images" paradigm in multimodal LLMs by conducting a causal audit of visual tool-use through a structured causal graph framework. The authors reveal that while visual operations like crop-and-zoom can yield marginal accuracy improvements, they often lead to miscalibrated policies and fail to provide causal benefits in many instances, resulting in two distinct failure modes. Their findings highlight that despite apparent aggregate gains, visual tool-use may not enhance model performance in a meaningful way, exposing a significant gap between perceived and actual efficacy.
Visual tool-use in multimodal LLMs may create an illusion of effectiveness, with many models achieving gains that are not causally justified.
The"thinking-with-images"paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.
