Search papers, labs, and topics across Lattice.
This paper introduces VistaVLA, a two-stage framework that enhances Vision-Language-Action (VLA) models for robotic manipulation by integrating geometry- and semantics-aware 3D representations derived from 3D Gaussian primitives. By employing a novel Merge-then-Query (MtQ) mechanism, VistaVLA compresses dense Gaussian data into compact context tokens, achieving a 99% reduction in token count while maintaining critical spatial and semantic information. The framework demonstrates significant improvements in task success rates, achieving a 22.8% increase in real-world scenarios and a 30.0% enhancement over existing baselines in challenging tasks.
A 99% reduction in token count without sacrificing spatial and semantic integrity leads to a 22.8% boost in real-world robotic task success rates.
Vision-Language-Action (VLA) models have emerged as a powerful end-to-end paradigm for robotic manipulation by mapping language instructions and 2D visual inputs directly to actions. However, these models lack an explicit, scene-level 3D representation, limiting their ability to reason over spatial layouts and geometric constraints. While recent efforts incorporate explicit 3D cues, such as depth maps or point clouds, to improve geometric awareness, they primarily capture low-level structures and lack high-level semantic grounding in 3D space. In human cognition, interaction with the physical world relies on a 3D semantic cognitive map - an internal mental model that integrates spatial layouts with semantic context to enable persistent, viewpoint-invariant reasoning. In light of this, we present VistaVLA, a novel two-stage framework that constructs a geometry- and semantics-aware 3D cognitive representation from 3D Gaussian primitives and grounds it as compact context tokens for VLA policy learning. Specifically, VistaVLA lifts multi-view vision-language features into 3D Gaussian primitives, forming geometry-anchored semantic tokens that align view-consistent spatial grounding with 2D visual feature spaces. To make this 3D representation computationally tractable for effective VLA control, we introduce Merge-then-Query (MtQ), a token summarization mechanism. MtQ compresses dense Gaussian primitives into a highly compact set of spatially informative tokens, achieving a 99% token reduction while preserving action-relevant 3D layouts and semantic context. Extensive evaluations in both simulated and real-world environments demonstrate the effectiveness of VistaVLA. Notably, in real-world scenarios, VistaVLA improves success rates by 22.8% across seven real-world tasks and by 30.0% over the VLA-Adapter baseline on challenging out-of-distribution tasks.
