Search papers, labs, and topics across Lattice.
This paper introduces Walking in the Implicit, a novel interactive video generation framework that utilizes a Neural Implicit Scene (NIS) to decouple state transitions from observation synthesis, enabling more efficient world exploration. By employing a transformer VAE to learn NIS from sparse posed frames and a diffusion transformer to evolve NIS based on future camera trajectories, the system achieves significant long-horizon consistency and inference efficiency. NeuWorld, the instantiated model, is trained from scratch without relying on pretrained video backbones, demonstrating strong performance in generating coherent video sequences over extended interactions.
NeuWorld achieves impressive long-horizon consistency in interactive video generation without relying on pretrained models or auxiliary 3D reconstruction methods.
Interactive video generation systems for camera-controlled world exploration roll out growing sequences of latent video frames, entangling state transition with high-frequency observation synthesis. We propose Walking in the Implicit, a scene-centric paradigm that changes the rollout variable from frame latents to a fixed-length, renderable implicit state, termed Neural Implicit Scene (NIS). This factorizes interactive generation into stochastic transition of a compact scene state and deterministic pose-conditioned rendering given the sampled state. We instantiate this paradigm as NeuWorld: a transformer VAE learns locally anchored NIS from sparse posed frames, and a diffusion transformer evolves NIS conditioned on future camera trajectories and geometry-aware retrieved history. By reusing the VAE encoder as a unified conditioner, NeuWorld maps camera, reference-image, and history cues into the same NIS modality, avoiding external heterogeneous encoders. Trained from scratch on public posed-view data without pretrained video backbones or auxiliary 3D reconstructors, NeuWorld achieves strong long-horizon consistency with favorable inference efficiency.
