Search papers, labs, and topics across Lattice.
This paper introduces a novel Riemannian Flow Matching framework that operates within the latent space of the Visual Geometry Grounded Transformer (VGGT), enabling the generation of plausible 3D geometry from unposed images. By respecting the latent geometry of VGGT, which occupies a product of high-dimensional hyperspheres, the authors overcome the limitations of traditional Euclidean flow matching. The proposed method demonstrates superior performance on datasets like RealEstate10K, ScanNet++, and ETH3D, effectively bridging the gap between deterministic geometric priors and generative 3D modeling.
Flow matching in latent spaces can unlock new capabilities for 3D scene generation, outperforming traditional methods by leveraging learned geometric priors.
Geometric foundation models, such as the Visual Geometry Grounded Transformer (VGGT), provide strong 3D priors from unposed images. However, such models operate purely in a feed-forward, deterministic regime, \ie~they cannot generate plausible geometry beyond what the input views directly support. Generative models for 3D scenes, on the other hand, must rely on strong geometric priors to produce coherent outputs from sparse inputs. We bridge these two paradigms by performing flow matching directly in VGGT's latent space, leveraging its learned 3D priors without committing to any explicit downstream representation such as Gaussians, meshes, or video-VAE latents. This requires respecting the latent geometry: VGGT tokens occupy a product of high-dimensional hyperspheres on which standard Euclidean flow matching fails. We address this with a Riemannian Flow Matching framework defined on a product manifold of four hyperspheres, aligned with VGGT's multi-scale encoder, which keeps generated tokens on the valid data manifold required by the frozen decoding heads. On RealEstate10K, ScanNet++ and ETH3D, our method achieves strong performance against recent scene generation baselines in both per-view appearance and aggregated 3D geometry, establishing latent-space flow matching on geometric foundation models as a viable paradigm for 3D generation. The project page can be found $\href{https://lisaweijler.github.io/geometry-grounded-rfm/}{\text{here}}$.
