Search papers, labs, and topics across Lattice.
This paper introduces ViDiHand, a novel approach for 4D hand motion reconstruction that utilizes pretrained video diffusion models to overcome the limitations of existing methods, which struggle with occlusion and insufficient training data. By adapting the diffusion model through a hand-overlay rendering objective, the authors effectively specialize its features for hand motion while maintaining world priors, allowing for accurate metric-scale pose recovery. The results demonstrate that ViDiHand significantly outperforms traditional techniques on multiple benchmark datasets, highlighting the potential of video diffusion models in this domain.
Video diffusion models can revolutionize hand motion reconstruction, achieving unprecedented accuracy without relying on traditional detection or optimization techniques.
4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.
