Search papers, labs, and topics across Lattice.
This paper introduces DexFuture, a hierarchical system designed to enhance bimanual dexterous tool use in robots by integrating a high-level Future-State Visuomotor Target Predictor with a low-level Target-Conditioned Structured Dexterous Policy. By leveraging egocentric RGB, proprioceptive, and geometric history, DexFuture generates structured visuomotor embeddings and a multi-step future target trajectory, enabling efficient tracking of complex hand-tool-object interactions. The approach achieves 90% of the performance of privileged-oracle methods while operating at 60 Hz, significantly outperforming traditional planning methods in both speed and effectiveness.
Achieving 90% of oracle performance in bimanual tool use, DexFuture operates 250 times faster than conventional planning methods.
Bimanual dexterous tool use remains challenging for robots due to high-dimensional hand configurations and complex hand-tool-object dynamics and contact. Most existing control policies depend on future configuration references provided from demonstrations, while future action-conditioned world models require slow online planning over high-dimensional action sequences. A significant challenge is generating a dynamically consistent future reference trajectory without relying on privileged states from demonstrations or slow counterfactual planning. We propose DexFuture, a hierarchical system that couples a high-level Future-State Visuomotor Target Predictor with a low-level Target-Conditioned Structured Dexterous Policy. Conditioned on egocentric RGB, proprioceptive and geometric history, the high-level predictor constructs structured hand-tool-object visuomotor embeddings and uses a horizon-conditioned transformer to generate a multi-step future target trajectory. Then, the low-level policy tracks them with a target-conditioned per-link transformer. This hierarchy decouples coarse future reference generation from fine-grained action control, and slow long-horizon semantic prediction from high-frequency execution. On OakInk2 bimanual tool-use tasks, DexFuture achieves 90% of the privileged-oracle performance, compared to 7% for a no-reference policy. DexFuture operates at 60 Hz, approximately 250 times faster than DexWM-style Cross-Entropy Method (CEM) planning with a future action-conditioned world model.
