Search papers, labs, and topics across Lattice.
This paper introduces a novel object-centric 3D representation alignment framework for Vision-Language-Action (VLA) models, leveraging SAM3D as a 3D teacher to enhance the understanding of target objects during robot manipulation tasks. By localizing task-relevant objects and aligning dense 3D representations with visual features, the approach maintains the original RGB-language-to-action pipeline without requiring complex 3D inputs at test time. The method shows significant performance improvements in both simulation and real-world scenarios, achieving a 99.1% success rate on LIBERO and excelling in long-horizon manipulation tasks.
Achieving 99.1% accuracy in object manipulation without the need for depth or additional 3D modules could revolutionize how robots interact with complex environments.
Vision-Language-Action (VLA) models have shown strong potential for general robot manipulation, but most existing models rely on 2D visual-language backbones and lack fine-grained 3D understanding of target objects, especially under occlusion, pose variation, scale changes, and precise spatial interaction. We propose an object-centric 3D representation alignment framework built upon $\pi_0$, using SAM3D as a frozen 3D teacher to provide target-object 3D priors during training. Specifically, we localize task-relevant objects with object recognition models, generate corresponding object masks, and use SAM3D to extract dense object-level 3D representations, which are aligned with intermediate visual features of $\pi_0$. This enables the policy to internalize target-object 3D information while preserving the original RGB-language-to-action inference pipeline without requiring depth, point clouds, masks, SAM3D, or additional 3D modules at test time. Simulation experiments show consistent improvements, achieving 99.1\% on LIBERO and an average length of 4.11 on CALVIN. Real-world experiments further demonstrate that our method is particularly effective in long-horizon manipulation scenarios where the robot must focus on different target objects across multiple subtasks.
