Search papers, labs, and topics across Lattice.
This paper introduces TeleDexter, a novel hand-object co-tracking controller designed to achieve human-level dexterous teleoperation by mapping operator intent into precise low-level contact execution. The controller is trained using a hybrid reward system that combines sparse subgoal objectives with dense tracking rewards, enabling effective learning across various interaction modalities without relying on frame-wise trajectory imitation. Evaluated across seven challenging tasks, TeleDexter achieved a 75% average success rate, significantly outperforming existing baselines and demonstrating its potential for training autonomous policies through behavioral cloning.
TeleDexter achieves a remarkable 75% success rate in dexterous teleoperation tasks, where existing systems fail, showcasing a leap towards human-level control in robotic manipulation.
Humans routinely wield tools, swap grasps, and reposition objects within a single hand, seamlessly orchestrating contact transitions that span translation, reorientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand-object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation. The entire pipeline requires only single-stage RL and, with random action masking and domain randomization, transfers zero-shot to the real robot. We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.
Tsinghua AI