Search papers, labs, and topics across Lattice.
HandsOnWorld introduces a novel framework for generating egocentric videos controlled by hand movements, eliminating the need for multi-view setups or marker-based motion capture. By leveraging monocular video and a unique protagonist-centered annotation pipeline, the authors create EgoVid-Pro, a comprehensive dataset featuring 3D hand trajectories from diverse real-world scenes. The proposed Pl眉cker Hand Map effectively disentangles camera and hand motion, resulting in superior reconstruction fidelity and control accuracy compared to existing methods, while also demonstrating generalization to out-of-distribution scenarios.
Unconstrained egocentric video generation now achieves unprecedented fidelity and control by disentangling hand and camera motion with a novel 3D-aware representation.
We present HandsOnWorld, a framework for hand-controlled egocentric video generation that forgoes multi-view and marker-based motion capture, learning instead from unconstrained monocular video. Such generality is bottlenecked by the scarcity of scalable 3D hand annotations: large egocentric corpora lack finger-level labels, whereas precise hand datasets are confined to narrow, instrumented settings, limiting prior hand-controlled generators to restricted scene distributions. We instead annotate 3D hands directly on in-the-wild egocentric video through monocular reconstruction, introducing a protagonist-centered annotation pipeline that filters the reconstructions at the action-semantic, image-quality, and 3D-geometric levels to build EgoVid-Pro, a dataset of clean, protagonist-only hand trajectories spanning 103K clips and roughly 12M frames across diverse everyday scenes. To resolve the camera-hand entanglement induced by large ego-motion, we further propose the Pl\"{u}cker Hand Map, a 3D-aware control signal that extends Pl\"{u}cker-ray representations from camera rays to the hand surface, disentangling camera and hand motion at the representation level. Experiments show that \method surpasses prior hand-controlled generators in reconstruction fidelity and control accuracy, and generalizes to out-of-distribution everyday scenes beyond the laboratory datasets on which prior methods rely.
Tsinghua AI