Search papers, labs, and topics across Lattice.
This paper introduces World from Motion, a novel method that generates dynamic 3D Gaussian representations from monocular video inputs by leveraging dense, pixel-aligned renderings to enhance reconstruction quality. By training on a dataset of aligned multiview video pairs and incorporating simulated monocular artifacts, the model effectively corrects rendering issues and fills in gaps in the initial reconstruction. The approach achieves state-of-the-art performance in 4D reconstruction, demonstrating robust generalization to diverse, real-world video scenarios with significant viewpoint changes and dynamic motion.
Achieving state-of-the-art 4D reconstruction, this method transforms monocular videos into high-quality dynamic 3D representations, even in challenging conditions.
We present World from Motion, a method for generating freely renderable dynamic 3D Gaussian representations from monocular videos. Our approach conditions a video model on dense, pixel-aligned renderings that encode appearance, geometry, and 3D scene motion along both input and target camera trajectories to correct rendering artifacts and fill in missing regions from an initial reconstruction. To train this model, we construct a dataset of aligned multiview video pairs and dynamic 3DGS representations, with simulated artifacts characteristic of monocular reconstruction. At test time, we distill the model's generations, including newly observed regions and motions, back into a single consistent, high-quality dynamic 3DGS, improving both novel-view synthesis and the underlying 3D motion. Our method sets a new state of the art in 4D reconstruction and seamlessly generalizes to in-the-wild videos with large viewpoint changes and dynamic motions.
Stanford HAI