Search papers, labs, and topics across Lattice.
This paper introduces OmniCoT, a comprehensive benchmark for enhancing the spatial reasoning capabilities of Multimodal Large Language Models (MLLMs) using panoramic imagery. By providing a structured training dataset with Chain-of-Thought annotations and a two-stage training strategy, OmniCoT enables MLLMs to perform complex global multi-step reasoning across 360掳 panoramic views. The results indicate that this approach significantly improves both answer accuracy and reasoning quality, addressing the limitations of existing benchmarks that focus on simplistic queries.
OmniCoT recalibrates the challenges of panoramic reasoning, enabling MLLMs to leverage global evidence for complex multi-step inference.
Multimodal Large Language Models (MLLMs) have demonstrated promising spatial reasoning capabilities, while these abilities remain underexplored in the emerging visual modality of panoramic imagery. The full 360掳$\times$180掳 field of view of panoramas essentially supports complex global multi-step reasoning, which is also the fundamental advantage of panoramas in applications such as embodied intelligence. However, existing panoramic benchmarks largely focus on simplistic queries that rely on local cues or single-/few-step reasoning, thereby ignoring the fundamental advantage of panoramas and failing to fully exploit their potential. To address this gap, we introduce OmniCoT, a panoramic spatial reasoning suite designed to enable MLLMs to use global evidence and perform multi-step inference across viewpoints. It includes OmniCoT-B (6.7K data) for evaluation, which measures both answer accuracy and reasoning quality, OmniCoT-Real (1K data) as a manually annotated real-world subset to quantify the Sim-to-Real gap. For training, OmniCoT-T (14.3K data) is purpose-built with structured stepwise Chain-of-Thought annotations that explicitly link intermediate reasoning steps to panoramic evidence. Based on OmniCoT-T, we introduce OmniCoT-R1 and adopt a two-stage training strategy tailored to the geometrically complex panoramic space, where Supervised Fine-tuning (SFT) anchors reasoning to panoramic evidence (e.g., bearings, proximity) and GRPO penalizes geometrically incoherent paths to consolidate global 360掳 spatial consistency. Through OmniCoT, we aim to recalibrate the difficulty of panoramic spatial reasoning to better align with the intrinsic capabilities of panoramic imagery, thereby fostering meaningful progress in this research area.
