Search papers, labs, and topics across Lattice.
This paper explores the use of internal representations from 3D Foundation Models (3DFMs) to infer 3D scene depth from novel viewpoints, addressing the challenge of 3D reconstruction. By leveraging the latent diffusion process on these representations, the authors introduce Z3D, a method that generates pointmaps for unseen views. The results demonstrate that Z3D can produce realistic depth maps across various datasets, highlighting the potential of 3DFMs in advancing 3D vision tasks.
Z3D reveals that 3D Foundation Models can effectively decode hidden surfaces to generate accurate depth maps for unseen views, transforming our approach to 3D reconstruction.
3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-foward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks. In this paper, we investigate using their internal representations to infer 3D in the scene from new views. Our hypothesis is that in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes. After showing that it is possible to decode hidden surfaces from internal 3DFM representations, we propose a method, Z3D, that estimates pointmaps in unseen views by doing latent diffusion on 3DFM representation. We show that Z3D can predict realistic depth maps for new views across multiple datasets.
