Search papers, labs, and topics across Lattice.
This paper introduces HOWTransfer, a novel hand-centric framework that effectively translates human video demonstrations into robotic trajectories by addressing challenges such as noisy hand-object interactions and unseen objects. By leveraging observed hand-object interaction cues, the method localizes contact intervals and generates diverse, executable robot motions that align with human grasp intent. Experimental results demonstrate an impressive 86% success rate in contact localization and superior performance compared to traditional teleoperation methods.
HOWTransfer achieves 86% success in translating human grasp intent into diverse, executable robot motions, outperforming teleoperation methods.
Learning from human video demonstrations remains challenging due to noisy hand-object interactions, unseen objects with partial observation, and cross-embodiment discrepancy. To address these challenges, we present \textit{HOWTransfer} (\emph{H}and-\emph{O}bject \emph{O}pen-\emph{W}orld Transfer), a hand-centric framework that distills human demonstrations into contact-aware, taxonomy-informed, and diverse robotic trajectories. Instead of relying on object-specific descriptions, vision-language queries, or explicit object-state tracking, \emph{HOWTransfer} recovers temporally consistent 3D hand motion and localizes temporal contact intervals by reasoning over observed hand-object interaction cues. The localized contact onsets are then used to retarget human grasp intent into multi-modal parallel-jaw grasp hypotheses, which are propagated along the recovered wrist trajectory to generate robot-executable motions. Finally, a trajectory editing stage refines contact alignment and produces diverse executable variants from a single demonstration. Experiments across diverse manipulation tasks show that \emph{HOWTransfer} enables accurate contact localization and high-quality robot motion retargeting with $86\%$ success, which is preferred over teleoperated trajectories in a blinded preference study.
