Search papers, labs, and topics across Lattice.
The paper introduces USR-Drive, a unified framework that simultaneously recovers dense dynamic geometry and instance-level object layouts from posed multi-view driving videos by treating 3D Gaussians and bounding boxes as aligned latent token streams. This approach addresses the limitations of existing methods that separate dynamic reconstruction from instance-level perception, leading to improved geometric grounding and spatial consistency. The results demonstrate state-of-the-art performance in both dynamic reconstruction and 3D detection tasks on the nuScenes and VKitti datasets, highlighting the effectiveness of the unified representation and generative framework.
A unified generative framework that simultaneously enhances dynamic geometry and instance-level detection achieves state-of-the-art results in autonomous driving scene representation.
Spatial representation learning for autonomous driving aims to map raw visual signals into structured 3D scene representations, where object-centric bounding boxes and rendering-oriented 3D primitives (\eg, 3D Gaussians) serve as two distinct yet highly complementary levels for scene understanding. Existing methods typically treat dynamic reconstruction and instance-level perception as separate tasks, despite their shared goal of estimating the underlying 3D world state. As a result, dynamic reconstruction is under-constrained while 3D detection lacks geometric grounding. To address this gap, we propose USR-Drive, a unified conditional generative framework that, given only posed multi-view driving videos, jointly recovers dense dynamic geometry and instance-level object layouts within a shared scene representation. Specifically, USR-Drive represents dense Gaussian primitives and sparse 3D bounding boxes as two aligned latent token streams and jointly denoises them with a unified multi-modal diffusion Transformer. Unlike prior paradigms that use boxes as external conditions or predict them with detached modules, USR-Drive treats them as mutually constrained state variables with a Unified Positional Encoding (UPE) that aligns heterogeneous tokens within a shared metric spatiotemporal coordinate. Via such unified representation and generative framework, the two modalities reinforce each other: geometry supplies dense metric evidence for box prediction, while boxes provide instance-level structural priors that help preserve spatial consistency and reduce ambiguity in sequential 3D geometric representation. Our approach successfully delivers state-of-the-art results for both dynamic reconstruction and 3D detection on the nuScenes and VKitti datasets.
