Search papers, labs, and topics across Lattice.
This paper introduces CRISP, a novel evaluation paradigm designed to diagnose visual spatial intelligence by assessing the consistency between implicit perception and explicit reasoning in visual language models (VLMs). The study reveals a critical disconnect between perception and reasoning capabilities, highlighting that while proprietary models exhibit strong latent reasoning, they struggle with metric estimation and underutilize their structural representations. In contrast, open-source models are hindered by their inability to perform multi-hop compositional reasoning, emphasizing the need for a shift towards genuine perception and reasoning in VLM evaluations.
Proprietary models may have powerful reasoning engines, but they fail to accurately estimate metrics and leverage structural insights, revealing a crucial gap in VLM performance.
Current VLM evaluations often conflate language priors with genuine spatial reasoning. To address this, we introduce CRISP, a novel structural-diagnostic evaluation paradigm that assesses visual spatial intelligence through consistency, the alignment between implicit perception and explicit reasoning. Unlike traditional black-box QA, CRISP utilizes metric 3D Scene Graphs and an oracle intervention protocol to decouple latent reasoning capabilities from perceptual bottlenecks. This granular diagnosis uncovers a systematic perception-reasoning disconnect. Crucially, we reveal that while proprietary models possess robust latent reasoning engines, they suffer from inaccurate metric estimation and a critical failure to leverage their implicit structural representations. Conversely, open-source models remain fundamentally bottlenecked by their lack of multi-hop compositional reasoning. By shifting the focus from merely ``guessing correctly'' via language priors to genuinely ``perceiving, verifying, and reasoning,'' CRISP offers a rigorous roadmap for multimodal alignment beyond end-to-end post-training. The code and dataset are available at https://github.com/iiyamayuki/CRISP-Bench.
