Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
DexAgent is an agentic Human2Sim2Robot pipeline that synthesizes diverse, physically grounded dexterous robot trajectories from a single egocentric human video and a language prompt. By pairing a four-stage generation workflow with property-specific physical verifiers and a self-evolving tool library, the system autonomously refines simulation reconstructions to handle complex articulated and deformable objects. Policies trained on DexAgent-generated, retextured data achieve a 3.5x higher success rate than baseline methods across eleven real-world manipulation tasks while progressively reducing synthesis overhead.
A single egocentric human video can yield real-world dexterous manipulation policies that beat existing baselines by 3.5x, unlocked by treating video-to-sim translation as a closed-loop, agentic tool-synthesis problem rather than a rigid heuristic pipeline.
Human videos offer a scalable source of demonstrations for dexterous robot manipulation. However, existing human-to-simulation-to-robot (Human2Sim2Robot) pipelines rely on predefined procedures that struggle to accommodate diverse object properties and interactions, particularly those involving articulated and deformable objects. We introduce DexAgent, an agentic Human2Sim2Robot framework that converts a single egocentric human video and a task prompt into physically grounded robot trajectories for policy training. It operates through four stages: semantic understanding of human videos, property-based simulation reconstruction, robot trajectory optimization, and robot data generation. At each stage, DexAgent adapts its approach to the task and object properties by selecting suitable skills from its tool library or developing new ones when needed. Property-specific verifiers assess stage outcomes for physical validity and task-specific requirements and provide feedback for refinement, preventing error propagation through the workflow. This adaptive, verification-guided process allows DexAgent to process diverse objects and long-horizon tasks. In the final stage, DexAgent varies object and robot states in simulation to generate diverse robot trajectories from a single human video, then retextures the rendered observations to facilitate sim-to-real transfer. Newly developed skills and verifiers are retained in its tool library, making it self-evolving to accumulate reusable capabilities. This reduces processing time as DexAgent encounters more human videos. Across eleven real-world tasks, policies trained with DexAgent-generated data achieve a 3.5x higher success rate than competing baselines. Project website: https://dexagent123.github.io/.
Stanford HAI