Search papers, labs, and topics across Lattice.
Seen2Scene introduces a visibility-guided flow matching approach trained directly on incomplete, real-world 3D scans for scene completion and generation, avoiding reliance on synthetic data. The method uses a sparse transformer operating on TSDF volumes encoded in sparse grids, with masking of unknown regions guided by visibility. Results show improved completion accuracy and generation quality compared to baselines by learning directly from real-world, incomplete 3D scans.
Real-world 3D scene completion is now possible without synthetic training data, thanks to visibility-guided flow matching that handles incomplete scans.
We present Seen2Scene, the first flow matching-based approach that trains directly on incomplete, real-world 3D scans for scene completion and generation. Unlike prior methods that rely on complete and hence synthetic 3D data, our approach introduces visibility-guided flow matching, which explicitly masks out unknown regions in real scans, enabling effective learning from real-world, partial observations. We represent 3D scenes using truncated signed distance field (TSDF) volumes encoded in sparse grids and employ a sparse transformer to efficiently model complex scene structures while masking unknown regions. We employ 3D layout boxes as an input conditioning signal, and our approach is flexibly adapted to various other inputs such as text or partial scans. By learning directly from real-world, incomplete 3D scans, Seen2Scene enables realistic 3D scene completion for complex, cluttered real environments. Experiments demonstrate that our model produces coherent, complete, and realistic 3D scenes, outperforming baselines in completion accuracy and generation quality.
