Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
Dex-One2Many addresses the generalization and exploration challenges in learning dexterous manipulation from single human videos by abstracting the demonstration into sequential scene graphs. These graphs guide reinforcement learning by generating diverse reset states and providing dense rewards, enabling efficient exploration and broad generalizability beyond demonstrated poses and grasps. The framework achieves state-of-the-art performance, outperforming baselines by 6.5% on seen configurations and a significant 71% on unseen scenarios across multiple tool-use tasks.
Abstracting human demonstrations into scene graphs unlocks zero-shot generalization for dexterous manipulation policies, achieving 71% better performance on unseen tasks than imitation-heavy methods.
While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.