Search papers, labs, and topics across Lattice.
PanoFuse addresses the field-of-view and occlusion vulnerabilities of perspective-only Vision-Language-Action (VLA) policies by augmenting local observations with global omnidirectional perception from a pretrained panoramic foundation model. To prevent heterogeneous feature interference, the architecture employs Decoupled Semantic-Geometric Routing (DSGR), isolating semantic and geometric representations into parallel streams that selectively inform state and action tokens via structured block-wise attention. Across seven real-world robotic manipulation benchmarks, the framework achieves an average task success rate of 52.9%, demonstrating marked generalization improvements under novel objects, unseen backgrounds, and heavy visual distractors.
Perspective cameras leave robot manipulation policies dangerously blind to occlusions and scene distractors鈥攁 vulnerability largely mitigated by routing decoupled panoramic geometry and semantics directly into the action expert.
Vision-Language-Action (VLA) policies have shown promising performance in language-conditioned robotic manipulation. However, most existing VLA systems rely on conventional perspective cameras with limited fields of view, often missing global scene context and leading to unreliable manipulation under visual occlusions, distractors, and unseen environments. In this work, we propose PanoFuse, a panorama-enhanced VLA framework that complements local manipulation observations with global panoramic perception. PanoFuse introduces a dedicated panoramic branch that leverages a pretrained panoramic foundation model to extract complementary semantic and geometric representations from omnidirectional observations. Rather than directly mixing these heterogeneous features, we introduce Decoupled Semantic-Geometric Routing (DSGR), which maintains semantic and geometric representations as separate context streams and selectively routes both to downstream state and action representations through structured block-wise attention. This design provides the action expert with global spatial context while preserving task-relevant semantic information from the pretrained VLA backbone. We further develop a synchronized data collection pipeline and construct a new real-world manipulation dataset containing panoramic RGB observations, wrist-view images, language instructions, robot states, and actions. Across seven evaluation settings, PanoFuse achieves an average success rate of 52.9%, outperforming the evaluated baselines and achieving consistent gains under novel-object, unseen-background, and distractor-rich settings. Code and data will be released publicly at https://xux-hnu.github.io/PanoFuse.
