Search papers, labs, and topics across Lattice.
The paper introduces Spherical Coordinate-based Positional Embedding (SoPE), a novel positional encoding method for 3D Large Vision-Language Models (LVLMs) that addresses the limitations of Rotary Position Embedding (RoPE) in capturing 3D spatial structures and angular dependencies. SoPE maps point cloud token indices into a 3D spherical coordinate space, enabling unified modeling of spatial locations and directional angles, and incorporates a multi-scale frequency mixing strategy for feature fusion. Experiments on 3D scene benchmarks demonstrate that SoPE enhances spatial awareness and improves performance compared to RoPE, with strong generalization capabilities shown in real-world deployments.
By encoding 3D point clouds in spherical coordinates, SoPE overcomes RoPE's limitations and significantly boosts spatial reasoning in 3D LVLMs.
3D Large Vision-Language Models (3D LVLMs) built upon Large Language Models (LLMs) have achieved remarkable progress across various multimodal tasks. However, their inherited position-dependent modeling mechanism, Rotary Position Embedding (RoPE), remains suboptimal for 3D multimodal understanding. The vanilla RoPE formulation fails to preserve essential three-dimensional spatial structures when encoding 3D tokens, and its relative distance computation overlooks angular dependencies, hindering the model's ability to capture directional variations in visual representations. To overcome these limitations, we introduce Spherical Coordinate-based Positional Embedding (SoPE). Our method maps point-cloud token indices into a 3D spherical coordinate space, enabling unified modeling of spatial locations and directional angles. This formulation preserves the inherent geometric structure of point-cloud data, enhances spatial awareness, and yields more consistent and expressive geometric representations for multimodal learning. In addition, we introduce a multi-scale frequency mixing strategy to fuse feature information across different frequency domains. Experimental results on multiple 3D scene benchmarks validate the effectiveness of our approach, while real-world deployment experiments further demonstrate its strong generalization capability.
