Search papers, labs, and topics across Lattice.
The paper introduces FROST-Drive, an end-to-end autonomous driving architecture that leverages a frozen vision encoder pretrained with a Vision-Language Model (VLM) to improve generalization. By freezing the encoder, the model retains the VLM's broad world knowledge, avoiding overfitting to driving-specific data. The architecture combines the frozen encoder with a transformer-based adapter and GRU-based decoder, and is trained with a custom loss function optimizing for Rater Feedback Score (RFS), achieving superior performance on the Waymo Open E2E Dataset compared to fully fine-tuned models.
Freezing a VLM's vision encoder and adapting it for end-to-end driving surprisingly beats fine-tuning, suggesting pre-trained general knowledge trumps domain-specific adaptation for robustness.
End-to-end (E2E) models in autonomous driving aim to directly map sensor inputs to control commands, but their ability to generalize to novel and complex scenarios remains a key challenge. The common practice of fully fine-tuning the vision encoder on driving datasets potentially limits its generalization by causing the model to specialize too heavily in the training data. This work challenges the necessity of this training paradigm. We propose FROST-Drive, a novel E2E architecture designed to preserve and leverage the powerful generalization capabilities of a pretrained vision encoder from a Vision-Language Model (VLM). By keeping the encoder's weights frozen, our approach directly transfers the rich, generalized world knowledge from the VLM to the driving task. Our model architecture combines this frozen encoder with a transformer-based adapter for multimodal fusion and a GRU-based decoder for smooth waypoint generation. Furthermore, we introduce a custom loss function designed to directly optimize for Rater Feedback Score (RFS), a metric that prioritizes robust trajectory planning. We conduct extensive experiments on Waymo Open E2E Dataset, a large-scale datasets deliberately curated to capture the long-tail scenarios, demonstrating that our frozen-encoder approach significantly outperforms models that employ full fine-tuning. Our results provide substantial evidence that preserving the broad knowledge of a capable VLM is a more effective strategy for achieving robust, generalizable driving performance than intensive domain-specific adaptation. This offers a new pathway for developing vision-based models that can better handle the complexities of real-world application domains.
