Search papers, labs, and topics across Lattice.
This paper introduces Dual Latent World Models (DLWM), a two-stage pre-training paradigm for Gaussian-centric autonomous driving. DLWM first reconstructs multi-view semantic and depth images to predict 3D Gaussians, then trains separate latent world models for occupancy forecasting and motion planning, guided by Gaussian flow and ego-planning respectively. Experiments on SurroundOcc and nuScenes show significant performance gains in 3D occupancy perception, 4D occupancy forecasting, and motion planning.
Pre-training with Dual Latent World Models unlocks significant performance gains in autonomous driving tasks by learning holistic Gaussian-centric representations.
Vision-based autonomous driving has gained much attention due to its low costs and excellent performance. Compared with dense BEV (Bird's Eye View) or sparse query models, Gaussian-centric method is a comprehensive yet sparse representation by describing scene with 3D semantic Gaussians. In this paper, we introduce DLWM, a novel paradigm with Dual Latent World Models specifically designed to enable holistic gaussian-centric pre-training in autonomous driving using two stages. In the first stage, DLWM predicts 3D Gaussians from queries by self-supervised reconstructing multi-view semantic and depth images. Equipped with fine-grained contextual features, in the second stage, two latent world models are trained separately for temporal feature learning, including Gaussian-flow-guided latent prediction for downstream occupancy perception and forecasting tasks, and ego-planning-guided latent prediction for motion planning. Extensive experiments in SurroundOcc and nuScenes benchmarks demonstrate that DLWM shows significant performance gains across Gaussian-centric 3D occupancy perception, 4D occupancy forecasting and motion planning tasks.
