Search papers, labs, and topics across Lattice.
This paper introduces RESELF, a unified framework that simultaneously reconstructs 3D scenes and estimates full-body motion from egocentric video, addressing the limitations of existing methods that treat these tasks independently. By leveraging a pre-trained geometry foundation model and employing a novel closed-loop kinematic feedback mechanism, RESELF effectively combines deterministic scene geometry reconstruction with generative motion inference. Experimental results demonstrate that RESELF significantly outperforms state-of-the-art approaches in depth estimation, camera tracking, and full-body motion estimation, showcasing its efficacy in joint recovery tasks.
RESELF achieves unprecedented accuracy in 3D scene reconstruction and motion estimation from egocentric video, outperforming traditional methods that tackle these challenges separately.
Complete 3D perception from egocentric video requires recovering the surrounding scene and the wearer's full-body motion in a shared metric frame. Existing methods typically address scene reconstruction and motion estimation separately: scene reconstruction methods ignore the wearer, whereas motion estimation methods lack explicit scene geometry and often depend on external trajectories. Joint recovery is challenging because the two tasks exhibit asymmetric visibility and require different prediction paradigms. The largely visible scene supports deterministic geometric regression, whereas the severely occluded body requires generative motion inference. We therefore propose RESELF (REconstructing the Scene and the sELF), a unified framework that couples deterministic metric geometry reconstruction with geometry-conditioned motion generation. RESELF adapts a geometry foundation model pre-trained on large-scale exocentric data to egocentric video using frame-wise scale and relative-pose consistency objectives. The resulting camera trajectory and latent geometric features condition a diffusion model that recovers the wearer's motion. A subsequent closed-loop kinematic feedback stage further refines the camera head while preserving the reconstructed scene geometry. To support training and evaluation, we curate EE4D-JSM from EgoExo4D by aligning egocentric video, sparse metric scene geometry, camera trajectories, and full-body motion annotations. Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks across depth estimation, camera tracking, and full-body motion estimation. Code, models, and datasets will be available at https://ka1guan.github.io/RESELF/.
