Search papers, labs, and topics across Lattice.
This paper addresses the limitations of end-to-end manipulation policies in robotics that rely on 2D RGB inputs by introducing a pixel-wise 3D representation called aligned vertex map and vertex spectrum, which incorporates 3D awareness into the training of Vision-Language Models (VLMs). The authors propose a Bird's-Eye-View (BEV) alignment framework that standardizes the spatial representation of inputs and outputs across various robot embodiments and camera setups, enhancing the robustness of manipulation policies. Their approach significantly improves the consistency and generalization of robotic actions in real-world environments, as demonstrated through a comprehensive data processing pipeline and a novel temporal alignment scheme for diverse trajectories.
A new 3D representation and BEV alignment framework drastically enhance the generalization of robotic manipulation policies across varied environments and robot types.
End-to-end manipulation policies, combined with web-scale pretrained Vision-Language Models (VLMs), show the promise for generalizable and dexterous robotic manipulation. However, they inherit two key limitations from 2D foundation models: 1) the reliance on 2D RGB inputs that ignores the intrinsically 3D nature of manipulation; and 2) the lack of spatial 3D alignment between input-output spaces as well as across diverse robot embodiments, camera setups, and trajectory datasets. In this paper, we present a series of contributions to address these issues. First, we introduce aligned vertex map and vertex spectrum -- a pixel-wise 3D representation that elevates 2D visual inputs to 3D, using camera calibration and optional depth. This novel input representation marries 3D awareness with the generalization of 2D large VLMs. Then, we propose to align the inputs and outputs of manipulation policies by expressing per-pixel 3D information of each camera view and robot actions to a shared coordinate. Based on this, we designate a canonical Bird's-Eye-View (BEV) alignment frame and innovatively propose to construct BEV images, producing a view-invariant representation robust to camera pose variations. To enable training and evaluation at scale, we develop a comprehensive data processing pipeline to perform such alignments; we also introduce a novel temporal alignment scheme for trajectories across diverse robots, human operators, and datasets. These contributions collectively mitigate input and output spatial-temporal misalignments, improving the consistency and generalization for real-world manipulation. Pretrained checkpoint, source code and data processing pipeline are available in https://hnuzhy.github.io/projects/Dex-BEV.
