Search papers, labs, and topics across Lattice.
This paper introduces PerceptDrive, a novel framework for end-to-end autonomous driving that leverages frozen perception foundation models and adaptive expert routing to enhance world-action modeling. By addressing the prior-to-plan transfer problem, PerceptDrive integrates teacher-distilled priors and dense observation latents, enabling expert-specific query branches to process scene information effectively. The framework achieves state-of-the-art performance on NAVSIM benchmarks, demonstrating that adaptive routing of perception priors significantly improves trajectory generation without the need for complex test-time candidate selection.
Adaptive routing of perception priors allows PerceptDrive to generate optimal driving trajectories in real-time without complex post-processing.
Frozen perception foundation models encode rich geometric, semantic, and dynamic knowledge. Yet narrow conditioning interfaces may attenuate task-relevant cues, while static fusion cannot adjust expert contributions to each scene. We cast this challenge as the prior-to-plan transfer problem and introduce PerceptDrive, a perception prior world-action modeling framework with adaptive expert routing. PerceptDrive feeds teacher-distilled priors from a frozen, driving-adapted provider and dense observation latents from a frozen self-supervised video encoder into a trainable expert-routed world-action model. Expert-specific query branches process these signals, while a prior-retention objective anchors each branch to its prior. A router predicts soft gates from a shared scene representation and combines the expert conditions before trajectory generation. During training, privileged rule-based sub-metric estimates for branch-specific trajectory drafts provide soft-gate distillation targets. The predicted action-free future latent conditions a flow-matching actor. At inference, privileged components are absent; with one front-facing camera, PerceptDrive generates one trajectory per planning step without test-time scoring, reranking, or search. Experiments show that PerceptDrive achieves state-of-the-art performance with 90.4 PDMS on NAVSIM v1 and 90.2 EPDMS on NAVSIM v2, outperforming existing methods. Ablations confirm complementary gains from prior retention and scene-conditioned routing, alongside differential reliance on the three priors. These results demonstrate that preserving and adaptively routing perception priors improves direct planning without test-time candidate selection.
Tsinghua AI