Search papers, labs, and topics across Lattice.
This paper introduces TerraTransfer, a novel approach for training end-to-end autonomous driving policies without relying on expert demonstrations by leveraging self-play in vectorized simulators. By decoupling the learning of driving from visual perception, the method significantly reduces the costs associated with data collection and labeling while enabling rapid training through millions of simulated rollout steps. The resulting policy not only matches but also exceeds the performance of existing end-to-end methods in photorealistic driving scenarios, showcasing a more efficient training paradigm.
Self-play in vectorized simulators allows TerraTransfer to achieve superior driving performance without the costly burden of expert data, reshaping the landscape of autonomous driving training.
End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all stages: collecting and labeling millions of driving frames is costly, and closed-loop RL on images is bottlenecked by the per-step cost of photorealistic rendering plus a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: millions of rollout steps per second, and a state distribution naturally rich in collisions, near-misses, and recoveries that no driving log contains. Our approach exploits this asymmetry by decoupling learning to drive from learning to see. We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone, through the action KL divergence and a batch-relational low-rank structural loss. The action target comes from the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.
