Search papers, labs, and topics across Lattice.
This paper introduces GaussianDWM++, a unified framework that integrates language-grounded reasoning, 3D scene understanding, and controllable 4D editing within Driving World Models (DWMs). By employing a foundation-feature Gaussian tokenizer and a geometry-aware Gaussian adapter, the model effectively aligns textual information with 3D scene structures, enabling fine-grained scene editing and multi-modal generation. Experimental results show that GaussianDWM++ achieves state-of-the-art performance across various benchmarks in scene understanding, visual grounding, and dynamic scene manipulation tasks.
Achieving state-of-the-art performance in scene understanding and controllable 4D generation, GaussianDWM++ redefines how we integrate language and 3D scene editing.
Driving World Models (DWMs) have recently advanced rapidly with generative models, yet most existing methods mainly focus on conditional scene generation and lack explicit 3D scene understanding, language-grounded reasoning, and controllable 4D editing capabilities. Moreover, commonly used point cloud, occupancy, or BEV representations make it difficult to achieve fine-grained alignment between textual information and the underlying 3D scene structure. To address these limitations, we propose a foundation-feature Gaussian driving world model that unifies scene understanding, language-grounded reasoning, controllable 4D editing, and multi-modal generation within a single framework. Specifically, we introduce a foundation-feature Gaussian tokenizer that directly distills Qwen/SigLIP visual-language features into 3D Gaussian primitives, building a compact open-vocabulary Gaussian semantic field. We further design a geometry-aware Gaussian adapter that combines importance-aware hierarchical selection with text-conditioned Perceiver-style cross-attention to aggregate dense Gaussian primitives into compact world tokens. To improve representation compatibility, we introduce a KL-based Gaussian--image distribution alignment objective that aligns Gaussian world tokens with foundation image tokens. Based on the aligned Gaussian representation, our framework further supports instruction-controllable scene editing, including weather-conditioned generation and dynamic vehicle manipulation. Extensive experiments on broader driving benchmarks demonstrate that our method achieves state-of-the-art performance across scene understanding, visual grounding, planning-oriented reasoning, and controllable 4D generation tasks. We will release the code and datasets publicly on Github.
