Search papers, labs, and topics across Lattice.
This paper introduces PointDiT, a minimalist pixel-space Diffusion Transformer that directly processes raw 3D point map patches, conditioned on image tokens from a pre-trained DINOv3 model. By eliminating the need for complex hybrid architectures and intricate loss functions, PointDiT achieves superior performance in single-image 3D reconstruction, outperforming existing latent diffusion models while maintaining simplicity. The key result demonstrates that PointDiT not only produces sharper geometric structures but also exhibits enhanced robustness in challenging scenarios, such as with transparent objects.
A minimalist approach to 3D reconstruction outperforms complex models, achieving sharper geometry and greater robustness in ambiguous scenarios.
State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.
