Search papers, labs, and topics across Lattice.
JoHan addresses the challenge of recovering accurate and temporally consistent 3D hand motion from video by proposing a unified generative framework that bypasses intermediate per-frame pose predictions. The model jointly generates aligned 2D and 3D local hand pose sequences, leveraging 2D visual cues to guide 3D motion reconstruction and a learned motion prior for temporal consistency. This approach achieves state-of-the-art accuracy and speed in both local hand-pose and camera-space reconstruction, while producing significantly smoother hand motion dynamics.
A single generative model recovers temporally consistent 3D hand motion from video by jointly learning 2D-3D pose dynamics, bypassing unreliable per-frame estimates.
Recovering faithful 3D hand motion from video remains challenging due to frequent occlusions and incomplete visual observations, which make frame-wise pose estimates unreliable and temporally inconsistent. To address this problem, we propose JoHan, a unified generative framework that recovers hand motion directly from video sequences without relying on intermediate per-frame pose predictions. Trained from scratch, our model jointly generates aligned 2D and 3D local hand pose sequences by learning their temporal dynamics and cross-representation correspondence. The generated 2D trajectories exploit direct spatial and temporal cues from the 2D images to guide the following generative 3D motion reconstruction, while the learned motion prior promotes temporal consistency. Their learned 2D-3D correspondence further enables recovery of the hand's global position and orientation relative to the camera. Extensive experiments on challenging benchmarks demonstrate significantly improved accuracy and speed in local hand-pose and camera-space reconstruction. Notably, our method captures much better hand-motion dynamics, producing significantly smoother motion than previous methods while maintaining high per-frame pose accuracy.
