Search papers, labs, and topics across Lattice.
This paper introduces SpatialCrafter, a two-stage framework for high-fidelity image-to-scene generation that overcomes limitations of existing video diffusion models by using a global 3D proxy. The framework consists of a Point-anchored Sparse Structure (PaSS) Flow module for generating geometrically consistent 3D proxies and a Generative Deferred Refiner for enhancing appearance details. Extensive experiments demonstrate that SpatialCrafter significantly reduces long-term drift and maintains robustness under challenging conditions, outperforming state-of-the-art methods in both synthetic and real-world scenarios.
SpatialCrafter achieves unprecedented 3D consistency in image-to-scene generation, effectively eliminating long-term drift and enhancing detail fidelity.
Explorable image-to-scene generation is essential for applications in gaming, robotics, and virtual reality. Existing methods based on video diffusion model (VDM) commonly rely on incomplete conditioning signals such as sparse point clouds or 2D panoramas, leading to stochastic hallucinations, long-term drifts and suboptimal 3D consistency. We present SpatialCrafter, a novel two-stage framework that addresses these issues by introducing a global 3D proxy for high-fidelity image-to-scene generation. Specifically, we decompose the generation process into global proxy generation and appearance refinement. For proxy generation, we propose a Point-anchored Sparse Structure~(PaSS) Flow module that predicts a spatially aligned and geometrically consistent 3D proxy. For appearance refinement, we re-frame the VDM as a Generative Deferred Refiner which synthesizes high-frequency photorealistic details upon proxy-defined scene geometry. To better integrate the proxy with the pre-trained VDM, we introduce Parallel Geometry Injection and Proxy-Aware Corruption training strategies, which improve robustness to proxy artifacts without disrupting the pretrained generative manifold. Furthermore, as no suitable dataset exists for this explorable scene generation task, we construct a new large-scale dataset of 115K scenes. To the best of our knowledge, it is the first hybrid dataset for image-to-scene generation. Extensive experiments on both synthetic and real-world datasets show that SpatialCrafter outperforms state-of-the-art methods, mitigates long-term drift, and remains robust and consistent under rapid camera motion and extreme viewpoint changes. Code, models, and the newly constructed dataset will be publicly released. See more at https://fangchuan.github.io/SpatialCrafter/.
