Search papers, labs, and topics across Lattice.
The paper introduces StudioRecon, a novel pipeline for 4D human-scene reconstruction that effectively utilizes sparse, low-overlap camera setups by decoupling the background from human subjects. This approach enhances output quality by synthesizing numerous camera-controlled views through a video diffusion model and robustly initializing human representations with cross-view identity association. The method achieves state-of-the-art results in novel view synthesis across multiple real-world datasets, enabling applications like novel trajectory rendering and human replacement while minimizing artifacts in under-observed regions.
StudioRecon transforms low-overlap camera captures into high-fidelity 4D reconstructions, overcoming significant limitations in existing methods.
Existing volumetric capture of dynamic human performance achieves high fidelity with dense camera arrays. However, in real-world scenarios, only a handful of low-overlap cameras are available, which degrades the output quality and leaves large areas unobserved. Recent 4D reconstruction methods have focused on low-overlap settings, yet they still produce noticeable artifacts in under-observed regions. Video diffusion models have emerged as another option, but they show geometrically inconsistent results for humans. To address these limitations, we propose StudioRecon, a pipeline that reconstructs 4D human scenes from sparse, low-overlap cameras by decoupling background and humans. We densify background supervision by synthesizing hundreds of camera-controlled novel views with a video diffusion model. We also robustly initialize deformable Gaussian humans with cross-view identity association and triangulated multi-view keypoint fitting. Finally, our recursive enhancement module with motion-adaptive consistency injection harmonizes the composed output, thereby further avoiding remaining artifacts. We achieve state-of-the-art novel view synthesis across four real-world datasets and demonstrate applications such as novel trajectory rendering and human replacement.
