Search papers, labs, and topics across Lattice.
SceneTransporter is introduced to address the problem of generating structured 3D scenes from a single image, where existing methods struggle to organize part-level 3D objects into distinct instances. The method reframes structured 3D scene generation as a global correlation assignment problem and solves an entropic Optimal Transport (OT) objective within the denoising loop of a compositional DiT model to enforce structural constraints. SceneTransporter achieves state-of-the-art performance in open-world scene generation by improving instance-level coherence and geometric fidelity through exclusive routing of image patches to 3D latents and encouraging the grouping of similar patches.
By using optimal transport to guide cross-attention, SceneTransporter disentangles image patches and 3D latents, leading to more coherent and geometrically faithful 3D scene generation from single images.
We introduce SceneTransporter, an end-to-end framework for structured 3D scene generation from a single image. While existing methods generate part-level 3D objects, they often fail to organize these parts into distinct instances in open-world scenes. Through a debiased clustering probe, we reveal a critical insight: this failure stems from the lack of structural constraints within the model's internal assignment mechanism. Based on this finding, we reframe the task of structured 3D scene generation as a global correlation assignment problem. To solve this, SceneTransporter formulates and solves an entropic Optimal Transport (OT) objective within the denoising loop of the compositional DiT model. This formulation imposes two powerful structural constraints. First, the resulting transport plan gates cross-attention to enforce an exclusive, one-to-one routing of image patches to part-level 3D latents, preventing entanglement. Second, the competitive nature of the transport encourages the grouping of similar patches, a process that is further regularized by an edge-based cost, to form coherent objects and prevent fragmentation. Extensive experiments show that SceneTransporter outperforms existing methods on open-world scene generation, significantly improving instance-level coherence and geometric fidelity. Code and models will be publicly available at https://2019epwl.github.io/SceneTransporter/.
CMU ML
Tsinghua AI