Search papers, labs, and topics across Lattice.
Microsoft ResearchCorresponding authorD-causal VAE adapted from Wan2.2-TIDrive. We further evaluate zero-shotTwenteJul 6, 2026arXiv:2607.05133The paper introduces UNIVERSE, a unified video-action model that leverages a mask-modulated Diffusion Transformer to enhance action generalization in autonomous driving by directly integrating future video prediction with ego-trajectory generation. By co-training video latents and trajectory tokens within shared parameters, the model effectively utilizes dense video supervision to improve trajectory denoising and cross-domain performance. Experimental results demonstrate that UNIVERSE outperforms existing architectures, achieving a 91.0 PDMS on NAVSIM and enabling strong zero-shot transfer to other datasets without fine-tuning.
UNIVERSE achieves a remarkable 4.3脳 speedup in trajectory inference while maintaining planning accuracy, revolutionizing how video dynamics inform autonomous driving actions.
World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality. However, it remains unclear which architecture better transfers video-modeling benefits to trajectory generation. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents future-target leakage and enables trajectory-only inference by removing future-video denoising at test time, achieving a $4.3\times$ speedup over joint video-action rollout while maintaining comparable planning accuracy. The same model also supports video-only and joint video-action rollouts. Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM (vs. 89.6 for the Two-DiT variant), and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while ablations confirm the importance of single-DiT unification, video co-training, and mask-based modality decoupling.
