Search papers, labs, and topics across Lattice.
This paper introduces SPAR3S, a sparse voxel-aligned 3D latent generative model that enables conditional scene completion from sparse multi-view images without the need for ground-truth 3D supervision. By leveraging photometric supervision through differentiable 3D Gaussian Splatting, the model learns to represent only occupied voxels in a structured latent space, allowing for efficient prediction of missing latent tokens and their spatial support. The results demonstrate that SPAR3S achieves superior novel-view quality on synthetic indoor scenes compared to existing methods and shows promising generalization capabilities on real-world datasets like RealEstate10k.
Achieving higher novel-view quality in 3D scene generation without requiring dense volumetric representations or ground-truth supervision is a game changer for practical applications.
Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.
CMU ML