Search papers, labs, and topics across Lattice.
This paper introduces a two-stage framework called AnyLift that utilizes 2D diffusion to reconstruct 3D human motion and human-object interactions (HOI) from Internet videos. The first stage synthesizes multi-view 2D motion data using keypoints extracted from videos, addressing the limitations of existing motion-capture datasets by incorporating underrepresented motion types. The second stage employs a camera-conditioned multi-view diffusion model to effectively recover 3D motion and HOI, demonstrating superior performance on complex motions and real-world interactions compared to previous methods.
AnyLift outperforms existing methods by accurately reconstructing 3D human motion and interactions from challenging Internet videos, including gymnastics and in-the-wild scenarios.
Reconstructing 3D human motion and human-object interactions (HOI) from Internet videos is a fundamental step toward building large-scale datasets of human behavior. Existing methods struggle to recover globally consistent 3D motion under dynamic cameras, especially for motion types underrepresented in current motion-capture datasets, and face additional difficulty recovering coherent human-object interactions in 3D. We introduce a two-stage framework leveraging 2D diffusion that reconstructs 3D human motion and HOI from Internet videos. In the first stage, we synthesize multi-view 2D motion data for each domain, leveraging 2D keypoints extracted from Internet videos to incorporate human motions that rarely appear in existing MoCap datasets. In the second stage, a camera-conditioned multi-view 2D motion diffusion model is trained on the domain-specific synthetic data to recover 3D human motion and 3D HOI in the world space. We demonstrate the effectiveness of our method on Internet videos featuring challenging motions such as gymnastics, as well as in-the-wild HOI videos, and show that it outperforms prior work in producing realistic human motion and human-object interaction.
