Search papers, labs, and topics across Lattice.
The paper introduces 3DVLA, a plug-and-play framework designed to enhance Vision-Language-Action (VLA) models by incorporating robust 3D scene understanding. 3DVLA addresses limitations in 3D spatial position extraction, instance understanding, and occlusion reasoning through multi-view consistent 3D feature encoding, instance estimation with high-level instance tokens, and a masked self-supervised 3D encoding branch. Experiments on LIBERO-Plus and RoboTwin 2.0 demonstrate that integrating 3DVLA with existing VLA baselines leads to significant improvements in robotic manipulation performance.
Injecting 3D scene understanding into VLAs without extra labels yields surprisingly large gains in robotic manipulation, suggesting a critical missing piece in current vision-language-action models.
Vision-Language-Action models have achieved remarkable progress in robotic manipulation, yet they suffer from a critical limitation: a lack of 3D scene understanding. This deficiency manifests as three intertwined challenges: weak extraction of 3D spatial positions without enforcing multi-view consistency, inadequate 3D instance understanding, and fragile reasoning under occlusion. Although mature 3D perception methods exist, their direct integration into VLA pipelines is hindered by architectural incompatibility and by heavy reliance on costly instance-level annotations. To address the above challenges, we propose 3DVLA, a plug-and-play framework that injects robust 3D reasoning into pretrained VLAs without requiring extra manual labels or discarding VLM priors. Specifically, 3DVLA tackles the three challenges through: (1) pervasive 3D feature encoding with explicit multi-view consistency constraints across all modalities and a Spatially-Conditioned Geometry Aggregation method, (2) an instance estimation module with high-level instance tokens for 3D instance awareness, and (3) a masked self-supervised 3D encoding branch that retains its predictor for visual token completion to handle occlusions. We integrate 3DVLA with multiple VLA baselines and evaluate on LIBERO-Plus and RoboTwin 2.0. Results show consistent and significant gains in manipulation performance, validating both the effectiveness and plug-and-play compatibility of our approach.
