Search papers, labs, and topics across Lattice.
This study explores the potential of large-scale dexterous grasp datasets to enhance functional dexterity in robotic tool use, moving beyond traditional grasp generation. By employing a hierarchical imitation learning framework that integrates high-level hand sub-goal prediction with a low-level goal-conditioned controller, the authors construct a substantial 355k-trajectory grasp-pretraining dataset. The results demonstrate a significant performance boost in both simulation and real-world scenarios, with a 33.3 percentage point increase in task success over existing methods, highlighting the versatility of grasp datasets for complex manipulation tasks.
Grasp datasets can revolutionize robotic dexterity, enabling significant improvements in articulated tool use performance.
Large-scale dexterous grasp datasets encode rich priors over hand-object interaction, but their use has largely been confined to grasp generation and pick-and-place manipulation. We study whether such data can instead support functional dexterity in articulated tool use, where a robot must acquire a tool, maintain contact, and operate its functional moving parts. We adapt a hierarchical imitation learning framework that combines high-level hand sub-goal prediction with a low-level goal-conditioned controller. We construct a 355k-trajectory grasp-pretraining dataset from large-scale dexterous grasp annotations and use it to pretrain the low-level controller. The controller is then fine-tuned on downstream task demonstrations. To evaluate this setting, we introduce DexCraft, a simulation benchmark with six articulated tool-use tasks requiring coordinated finger motion. Across simulation and real-world experiments, our approach outperforms end-to-end diffusion policy baselines and hierarchical policies trained from scratch. In the real world, it improves full-task success by 33.3 percentage points over DP3. These results show that grasp datasets can serve not only as resources for grasp synthesis, but also as scalable pretraining data for contact-rich dexterous manipulation. Videos are shown on https://yingyuan0414.github.io/grasp2dexterity/ .
CMU ML