Search papers, labs, and topics across Lattice.
The paper introduces WorldStereo, a framework that leverages global-geometric and spatial-stereo memory modules to generate multi-view consistent videos with precise camera control, enabling high-quality 3D reconstruction. The global-geometric memory uses incrementally updated point clouds for camera control and coarse structural priors, while the spatial-stereo memory constrains attention with 3D correspondence for fine-grained details. Experiments demonstrate the framework's effectiveness in camera-guided video generation and 3D reconstruction, showcasing its capability as a world model for diverse scene generation tasks.
Generate multi-view consistent videos with precise camera control and reconstruct high-quality 3D scenes from them, all without joint training.
Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model's attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.
