Search papers, labs, and topics across Lattice.
HandFlow introduces a fully generative flow-matching framework for 4D hand reconstruction from monocular video, addressing the limitations of traditional deterministic models that struggle with occlusion and motion blur. By leveraging a dual-stream transformer and a confidence-aware continuous masking mechanism, it effectively denoises hand pose and shape estimates across entire temporal windows, resulting in significant improvements in both accuracy and temporal coherence. Experiments demonstrate that HandFlow achieves over 30% reduction in world-space pose error and operates at 47 fps, outperforming existing methods in speed and accuracy.
HandFlow reduces world-space pose error by over 30% while reconstructing 150 frames at 47 fps, setting a new standard for 4D hand recovery.
Accurate monocular 4D hand reconstruction remains challenging. Per-frame discriminative regressors lack temporal context and often produce jittery predictions. Temporal models improve consistency by aggregating information across frames, but they are typically deterministic regressors, making them vulnerable to ambiguous observations caused by occlusion and motion blur. Generative modeling offers a natural alternative by learning a prior over plausible hand motion sequences, enabling coherent hand-state recovery when visual evidence is incomplete or unreliable. Motivated by this observation, we present HandFlow, a fully generative flow-matching framework for temporally coherent 3D hand pose and shape estimation from monocular video. Given visual and skeletal observations, HandFlow denoises an entire temporal window of MANO parameters through a single ODE integration. To support this, we use a Flux-style dual-stream transformer that attends across the full sequence to capture long-range dependencies without autoregressive decoding, and a confidence-aware continuous masking mechanism that blends observed features with learnable mask tokens to handle noisy or missing observations. Experiments on DexYCB and HOT3D show that HandFlow achieves state-of-the-art performance, with particularly large gains in world-space accuracy and temporal smoothness. It reduces world-space pose error by over 30% compared with the strongest baseline and achieves the lowest acceleration error among all evaluated methods, while remaining competitive in per-frame pose accuracy. Moreover, on a single GPU HandFlow reconstructs a 150-frame sequence at 47 fps, about 12x faster than the fastest prior video-based method, with reconstruction itself accounting for only a small fraction of the end-to-end latency.
Google Research