Search papers, labs, and topics across Lattice.
This paper introduces 3D HAMSTER, a hierarchical framework that enhances Vision-Language-Action (VLA) models by generating metrically reliable 3D trajectories for robotic manipulation. By integrating a depth encoder and a dense depth reconstruction objective into the planning process, the model overcomes the limitations of 2D trajectory guidance, which can lead to significant geometric distortions. Experimental results demonstrate that 3D HAMSTER outperforms existing VLMs and 2D-guided approaches, particularly in challenging scenarios involving appearance alterations and novel language inputs.
3D HAMSTER achieves superior robotic manipulation performance by directly predicting 3D trajectories, eliminating the distortions caused by 2D guidance.
Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation. Recent work in this paradigm uses 2D end-effector trajectories predicted by a Vision-Language Model (VLM) as explicit guidance for a downstream policy. However, state-of-the-art low-level policies operate in 3D metric space on point clouds, and feeding them 2D guidance that lacks depth forces each waypoint to be assigned the depth of whatever scene surface lies beneath it, producing geometrically distorted trajectories. We propose 3D HAMSTER, a hierarchical framework that closes this gap by having the planner directly output metrically reliable 3D trajectories. We augment a VLM with a dedicated depth encoder and a dense depth reconstruction objective to predict 3D waypoint sequences, which are directly integrated into a pointcloudbased low-level policy. Across 3D trajectory prediction, simulation, and real-world manipulation, 3D HAMSTER consistently outperforms proprietary VLMs and 2D-guided baselines, with the largest gains under appearance-altering shifts and unseen language, spatial, and visual conditions. The project page is available at https://davian-robotics.github.io/3D_HAMSTER/.
