Search papers, labs, and topics across Lattice.
UMI-3D enhances the Universal Manipulation Interface (UMI) by integrating a low-cost LiDAR sensor for robust 3D spatial perception, overcoming limitations of the original vision-only UMI in challenging environments. This LiDAR-centric SLAM system provides accurate metric-scale pose estimation and is combined with a hardware-synchronized multimodal sensing pipeline and spatiotemporal calibration framework to align visual and LiDAR data. Experiments show UMI-3D significantly improves data quality and enables learning of manipulation tasks previously infeasible with vision-only UMI, such as large deformable object manipulation.
Vision-based robot learning struggles with occlusions, but UMI-3D's clever wrist-mounted LiDAR unlocks robust manipulation in cluttered real-world environments.
We present UMI-3D, a multimodal extension of the Universal Manipulation Interface (UMI) for robust and scalable data collection in embodied manipulation. While UMI enables portable, wrist-mounted data acquisition, its reliance on monocular visual SLAM makes it vulnerable to occlusions, dynamic scenes, and tracking failures, limiting its applicability in real-world environments. UMI-3D addresses these limitations by introducing a lightweight and low-cost LiDAR sensor tightly integrated into the wrist-mounted interface, enabling LiDAR-centric SLAM with accurate metric-scale pose estimation under challenging conditions. We further develop a hardware-synchronized multimodal sensing pipeline and a unified spatiotemporal calibration framework that aligns visual observations with LiDAR point clouds, producing consistent 3D representations of demonstrations. Despite maintaining the original 2D visuomotor policy formulation, UMI-3D significantly improves the quality and reliability of collected data, which directly translates into enhanced policy performance. Extensive real-world experiments demonstrate that UMI-3D not only achieves high success rates on standard manipulation tasks, but also enables learning of tasks that are challenging or infeasible for the original vision-only UMI setup, including large deformable object manipulation and articulated object operation. The system supports an end-to-end pipeline for data acquisition, alignment, training, and deployment, while preserving the portability and accessibility of the original UMI. All hardware and software components are open-sourced to facilitate large-scale data collection and accelerate research in embodied intelligence: \href{https://umi-3d.github.io}{https://umi-3d.github.io}.
