Search papers, labs, and topics across Lattice.
This paper introduces DreamHand, a novel framework that repurposes video diffusion models (VDMs) to recover occlusion-robust 3D hand trajectories from egocentric video. By utilizing a Deterministic Clean-Latent Encoder and a Bidirectional Spatiotemporal Decoder, DreamHand effectively captures continuous bimanual motion without reliance on external detectors, achieving significant improvements in accuracy. The method sets new benchmarks in hand motion recovery, reducing mean per joint position error (MPJPE) by up to 40% on occlusion-heavy datasets, thus enhancing the scalability of using everyday human video for robotic manipulation tasks.
DreamHand achieves a groundbreaking 40% reduction in error for 3D hand trajectory recovery in occluded environments, setting a new standard for egocentric video analysis.
Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-sight gaps. Existing single-frame and windowed temporal regressors fail when hand shortly leaves the frame, while recent video diffusion models (VDMs) rely on heavy, stochastic multi-step sampling as pixel-space renderers. We instead repurpose VDM into a deterministic geometry encoder. A single forward pass over the clean latent exposes scene content beyond current observations, including occluded and out-of-sight hands. We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent Encoder and decodes them with a Bidirectional Spatiotemporal Decoder. DreamHand recovers continuous bimanual trajectories with metric placement and no external detector, while a Ray-Based Camera Solver supports a second configuration that needs no test-time camera intrinsics. Across five egocentric benchmarks, DreamHand sets a new state of the art, cutting MPJPE-p by 30% on occlusion-heavy ARCTIC and 40% on HOT3D. These gains reach 46%-61% once out-of-sight hands are included in the evaluation, offering a scalable path from everyday human video to robot manipulation data.
