Search papers, labs, and topics across Lattice.
SocialMirror, a diffusion-based framework, reconstructs 3D human interaction behaviors from monocular videos by addressing challenges like occlusion and motion ambiguity. It uses high-level interaction descriptions from a vision-language model to guide a semantic-guided motion infiller, hallucinating occluded bodies and resolving pose ambiguities. A sequence-level temporal refiner then enforces smooth motions and geometric constraints, ensuring plausible contact and spatial relationships.
Diffusion models, guided by vision-language models and geometric constraints, can now reconstruct realistic 3D human interactions from monocular video, even with severe occlusions.
Accurately reconstructing human behavior in close-interaction scenarios is crucial for enabling realistic virtual interactions in augmented reality, precise motion analysis in sports, and natural collaborative behavior in human-robot tasks. Reliable reconstruction in these contexts significantly enhances the realism and effectiveness of AI-driven interactive applications. However, human reconstruction from monocular videos in close-interaction scenarios remains challenging due to severe mutual occlusions, leading local motion ambiguity, disrupted temporal continuity and spatial relationship error. In this paper, we propose SocialMirror, a diffusion-based framework that integrates semantic and geometric cues to effectively address these issues. Specifically, we first leverage high-level interaction descriptions generated by a vision-language model to guide a semantic-guided motion infiller, hallucinating occluded bodies and resolving local pose ambiguities. Next, we propose a sequence-level temporal refiner that enforces smooth, jitter-free motions, while incorporating geometric constraints during sampling to ensure plausible contact and spatial relationships. Evaluations on multiple interaction benchmarks show that SocialMirror achieves state-of-the-art performance in reconstructing interactive human meshes, demonstrating strong generalization across unseen datasets and in-the-wild scenarios. The code will be released upon publication.
