Search papers, labs, and topics across Lattice.
This paper introduces Hand-4DGS, a novel feed-forward framework designed for dynamic 4D hand reconstruction from egocentric videos, addressing challenges such as rapid hand dynamics and severe occlusions. The method leverages a mesh-guided representation and temporal convolutions to achieve fast inference at approximately 60 FPS while maintaining strong generalization capabilities. Evaluations on the H2O and ARCTIC datasets show that Hand-4DGS significantly outperforms existing baselines, demonstrating its effectiveness in real-world applications for AR/VR and AI glasses.
Achieving 60 FPS dynamic 4D hand reconstruction from egocentric videos, Hand-4DGS outperforms traditional methods by effectively handling occlusions and rapid motion.
Dynamic 3D hand reconstruction from egocentric videos is essential for next-generation computing platforms such as AR/VR and AI glasses. Despite its importance, most prior works focus either on multi-view 3D hand reconstruction or on 4D human body reconstruction. Egocentric 4D hand reconstruction remains challenging due to fast head motion, rapid hand dynamics, severe occlusions, and inherent ambiguity from single-view observations. To address these challenges, we introduce Hand-4DGS, the first feed-forward framework for reconstructing dynamic 4D hands directly from egocentric videos, enabling both fast (~60 FPS) inference and strong generalization. Our approach incorporates a mesh-guided representation for structural priors and temporal convolutions to model dynamic motion. We evaluate our framework on two challenging egocentric datasets, H2O and ARCTIC, and demonstrate significant improvements over baselines. Our method benefits from the generalization capability of feed-forward networks and effective 2D image supervision through Gaussian splatting, without requiring expensive 3D hand pose ground-truth annotations.
ETH
Microsoft Research