Search papers, labs, and topics across Lattice.
This paper addresses the limitations of multimodal large language models (MLLMs) in spatial reasoning, particularly their reliance on camera-centric cues that lead to errors in non-camera reference contexts. The authors introduce OrientSAM, an innovative framework that incorporates orientation-aware tokens and Fourier-based angle encoding to enhance perspective-aware reasoning through explicit orientation modeling. Experimental results across multiple datasets demonstrate that OrientSAM significantly outperforms existing baselines, highlighting the importance of orientation in achieving robust allocentric spatial reasoning.
Explicitly modeling object orientation can drastically reduce systematic errors in multimodal spatial reasoning, outperforming traditional methods by a significant margin.
Multimodal large language models (MLLMs) still struggle with spatial reasoning that requires perspective transformation. In particular, they often rely on camera-centric cues rather than reasoning from the reference object's viewpoint, leading to systematic errors in non-camera reference settings. In this paper, we first analyze this failure mode and show that object orientation is a key factor underlying such camera-centric shortcut behavior. To address this issue, we propose OrientSAM, an orientation-aware spatial alignment framework for multimodal models. OrientSAM injects explicit orientation information into multimodal representations through orientation-aware tokens and Fourier-based angle encoding, and further adopts a curriculum learning strategy to progressively improve perspective-aware reasoning. In addition, we build a spatial data construction pipeline to generate orientation-aware spatial supervision from large-scale images. Experiments on Spatial-MM, ViewSpatial, and 3DSRBench show that OrientSAM consistently outperforms strong baselines, especially on non-camera-view, person-centric, and orientation-sensitive tasks. The results further demonstrate that explicit orientation modeling is important for mitigating camera-centric shortcut behavior and enabling more robust allocentric spatial reasoning in multimodal models.
