Search papers, labs, and topics across Lattice.
OmniUMI is a novel handheld interface that captures synchronized RGB, depth, trajectory, tactile sensing, grasping force, and external wrench data for physically grounded robot learning. By providing dual-force feedback, the system enables human-aligned demonstrations for contact-rich manipulation tasks. Experiments show that diffusion policies trained with OmniUMI data achieve strong performance in force-sensitive pick-and-place, surface erasing, and selective release tasks.
Robots can now learn contact-rich manipulation skills like humans by feeling the forces involved, thanks to a new multimodal interface that captures synchronized visual, tactile, and force data.
UMI-style interfaces enable scalable robot learning, but existing systems remain largely visuomotor, relying primarily on RGB observations and trajectory while providing only limited access to physical interaction signals. This becomes a fundamental limitation in contact-rich manipulation, where success depends on contact dynamics such as tactile interaction, internal grasping force, and external interaction wrench that are difficult to infer from vision alone. We present OmniUMI, a unified framework for physically grounded robot learning via human-aligned multimodal interaction. OmniUMI synchronously captures RGB, depth, trajectory, tactile sensing, internal grasping force, and external interaction wrench within a compact handheld system, while maintaining collection--deployment consistency through a shared embodiment design. To support human-aligned demonstration, OmniUMI provides dual-force feedback through bilateral gripper feedback and natural perception of external interaction wrench in the handheld embodiment. Built on this interface, we extend diffusion policy with visual, tactile, and force-related observations, and deploy the learned policy through impedance-based execution for unified regulation of motion and contact behavior. Experiments demonstrate reliable sensing and strong downstream performance on force-sensitive pick-and-place, interactive surface erasing, and tactile-informed selective release. Overall, OmniUMI combines physically grounded multimodal data acquisition with human-aligned interaction, providing a scalable foundation for learning contact-rich manipulation.
