Search papers, labs, and topics across Lattice.
This paper introduces a compact latent world model that predicts future camera scenes for autonomous vehicles based on ego-actions, addressing the limitations of standard distortion metrics that favor blurry predictions. By benchmarking various encoders, it finds that the V-JEPA2 model with temporal context significantly reduces steering RMSE by 40% compared to the best single-frame encoder. The authors develop a Diffusion Transformer that achieves superior scene prediction quality, evidenced by a KID score of 0.078, outperforming traditional regression methods by a factor of 4.8, while maintaining action-controllability in scene displacement.
Action-conditioned predictions from a compact latent model reveal that diffusion methods can dramatically outperform traditional regression in scene forecasting for autonomous vehicles.
Action-conditioned world models let an autonomous vehicle predict future camera scenes from its own planned controls, enabling planning and simulation without real-world rollouts, but at compact, trainable scale the futures are ambiguous and the field's standard distortion metrics actively mislead: they reward a blurry regression mean over a realistic prediction. We confront this with a compact latent world model that, given the present front-camera latent and a sequence of ego-actions, predicts future scene latents a frozen decoder renders to $256 \times 256$ frames up to 8 seconds ahead, evaluated on 150 held-out nuScenes scenes. We first benchmark where to predict: across six frozen encoders spanning four representation families, V-JEPA2 with temporal context reduces steering RMSE by 40% over the best single-frame encoder. We then train a latent Diffusion Transformer (DiT) and, through a controlled diagnosis, identify the four ingredients it needs: spatial tokens, the $x_0$ objective, residual anchoring, and sampling matched to target uncertainty. In a Stable-Diffusion-VAE encode-predict-decode pipeline we expose the central tension: distortion metrics (cosine similarity, SSIM) favor the blurry mean, masking that the diffusion model is far closer to the real frame distribution. Inception-based FID and KID reveal a clean perception-distortion frontier: diffusion attains KID 0.078 versus 0.375 for regression ($4.8\times$ better), and a deployable train-derived calibration makes this practical without test-time ground truth. The model is genuinely action-controllable (steering drives scene displacement, Spearman $\rho = 0.81$, vs $-0.18$ for regression). We trace limited single-pass motion to a shared-present anchor and engineer a compact 1.7M-parameter"jump"model that recovers full ground-truth motion magnitude ($1.02\times$ GT), where single-pass models capture less than half.
Stanford HAI