Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper introduces EMPIRE, a two-stage framework for forecasting dexterous hand motions from egocentric observations by integrating Explicit Manipulation Planning as an intermediate representation. The first stage focuses on learning manipulation plans from multimodal contexts to effectively capture hand-object interactions, while the second stage synthesizes future bimanual motions based on these frozen plans, mitigating interference from motion-generation gradients. EMPIRE achieves state-of-the-art forecasting accuracy on a newly constructed dataset, EMPIRE-651K, demonstrating significant improvements over existing methods with an MPJPE of 84.53 mm and a finger-relative error of 38.97 mm.
Explicit manipulation planning can dramatically enhance the accuracy of hand-motion forecasting, achieving state-of-the-art results without interference from motion synthesis.
Forecasting dexterous hand motions from egocentric observations is fundamental to intelligent interactive systems. Existing VLM-based methods typically map observations directly to future motions, overlooking the underlying manipulation process that governs hand-object interactions. Moreover, end-to-end optimization couples manipulation learning with motion synthesis, causing motion-generation gradients to interfere with the pre-learned manipulation-aware representations. To overcome these limitations, we propose EMPIRE, a two-stage framework that introduces Explicit Manipulation Planning as an Intermediate Representation for Egocentric hand-motion forecasting. Stage I: Learn to Plan. EMPIRE first learns explicit manipulation plans from multimodal context to capture the progression of hand-object interactions. Stage II: Learn to Act. A motion generator synthesizes future bimanual hand motions conditioned on frozen planner representations, preventing motion-generation gradients from affecting manipulation planning. To support our method, we further construct EMPIRE-651K, a bimanual hand-motion forecasting dataset comprising 650,910 training windows across 111 tasks, each paired with an explicit per-hand manipulation plan. Under identical training and evaluation protocols, EMPIRE achieves state-of-the-art forecasting accuracy, with an MPJPE of 84.53 mm and a finger-relative error of 38.97mm. We release the code and dataset at https://github.com/wangwen-banban/EMPIRE.