Search papers, labs, and topics across Lattice.
To bridge the gap in transferring complex 3D human hand motions to robotic grippers, the authors construct CosmoH2G鈥攁 paired dataset of 6,189 demonstrations across 1,254 objects鈥攁nd establish a robust two-stage transfer framework. Directly predicting full continuous gripper trajectories from hand poses typically causes severe compounding drift during intricate rotations and flips, which the authors resolve by first predicting sparse boundary keyframes and then generating conditioned intermediate trajectories. In both simulation and real-robot benchmarks, this approach combined with kinematic post-optimization substantially outperforms traditional end-to-end baselines on high-degree-of-freedom spatial manipulations.
Human-to-robot demonstration transfer notoriously collapses when tasks leave the 2D plane, but decoupling boundary keyframes from dense trajectory generation unlocks precise 3D flips and rotations across more than 1,200 objects.
Transferring human hand demonstrations to robotic grippers has recently emerged as a cost-effective solution for robot learning. However, existing methods are largely confined to simple, planar tasks and fail to handle complex spatial movements (e.g., intricate trajectories involving rotations or flips) that are essential for robot manipulation. Motivated by this gap, we adopt an implicit, data-driven approach guided by fine-grained hand-pose motions. To this end, we introduce a scalable acquisition pipeline to collect hand-gripper paired demonstrations, governed by a rigorous protocol that prioritizes motion complexity and leverages a handheld gripper for seamless action mimicry. This yields a large-scale paired dataset comprising 6,189 episodes across 1,254 unique objects, exhibiting significantly higher spatial complexity than existing benchmarks. However, learning such complex mappings remains challenging. We observe that naive end-to-end generation of full gripper pose sequences is insufficient, as minor trajectory deviations compound rapidly under intricate dynamics. To address this, we propose a two-stage framework: Stage I predicts sparse gripper keyframes (initial and terminal) to simplify the mapping objective, while Stage II generates the full continuous action sequence conditioned on these keyframes. Furthermore, to mitigate cumulative drift, we keep the gripper's orientation being learned while post-optimizing its translation based on the grasping heuristic and kinematic consistency. In both simulation and real-robot experiments, our framework enables stable and precise hand-to-gripper transfer of complex spatial manipulations, significantly outperforming traditional baselines. Project page: https://cosmoh2g.github.io.
