Search papers, labs, and topics across Lattice.
This paper introduces the Point as Skeleton framework, which enhances autoregressive video generation for closed-loop autonomous driving simulations by integrating generative sensor simulation with step-wise updated states. The method employs a Reset-and-Roll technique to manage latent states across simulation steps, while point-cloud skeletons are utilized to stabilize error accumulation by separating foreground and background elements. Experimental results on nuScenes and nuPlan datasets reveal significant improvements in the visual fidelity of generated driving simulations, highlighting the framework's effectiveness in maintaining realism during closed-loop interactions.
Generating visually faithful driving simulations just got a boost with a novel framework that stabilizes error accumulation and enhances realism in closed-loop scenarios.
Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity (e.g., CARLA) and real-world visual fidelity (e.g., nuScenes). We present \textbf{\emph{Point as Skeleton}}, a generative sensor simulation framework for state-updated autoregressive driving video generation, in which an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. To support closed-loop rollout, we introduce Reset-and-Roll, which adapts rolling diffusion inference to simulation by preventing future-conditioned latent states from being committed across simulation steps. To stabilize error accumulation during step-wise autoregressive rollout, we introduce point-cloud skeletons that decouple foreground and background assets and project them into camera-view painted-point and template-depth conditions, providing appearance and geometric cues. We further implement a nuPlan-based renderer-level closed-loop generative interface for evaluating generation under ego deviations from the original log. Experiments on nuScenes and nuPlan show that \textit{Point as Skeleton} improves autoregressive generation quality during closed-loop rollout, demonstrating its potential for visually faithful closed-loop driving simulation. The code is available at https://github.com/krauwu/point-as-skeleton.
