Search papers, labs, and topics across Lattice.
The paper introduces WHOLE, a method for jointly reconstructing hand and object motion in world space from egocentric videos, addressing limitations of existing methods that treat hand and object pose estimation in isolation. WHOLE learns a generative prior over hand-object motion to jointly reason about their interactions, enabling robust reconstruction even with occlusions and out-of-sight objects. Experiments demonstrate that WHOLE outperforms existing methods on hand motion estimation, 6D object pose estimation, and relative interaction reconstruction.
By jointly modeling hand-object motion, WHOLE overcomes occlusion and out-of-sight challenges in egocentric videos, achieving state-of-the-art results in hand and object pose estimation.
Egocentric manipulation videos are highly challenging due to severe occlusions during interactions and frequent object entries and exits from the camera view as the person moves. Current methods typically focus on recovering either hand or object pose in isolation, but both struggle during interactions and fail to handle out-of-sight cases. Moreover, their independent predictions often lead to inconsistent hand-object relations. We introduce WHOLE, a method that holistically reconstructs hand and object motion in world space from egocentric videos given object templates. Our key insight is to learn a generative prior over hand-object motion to jointly reason about their interactions. At test time, the pretrained prior is guided to generate trajectories that conform to the video observations. This joint generative reconstruction substantially outperforms approaches that process hands and objects separately followed by post-processing. WHOLE achieves state-of-the-art performance on hand motion estimation, 6D object pose estimation, and their relative interaction reconstruction. Project website: https://judyye.github.io/whole-www
