Search papers, labs, and topics across Lattice.
This paper investigates catastrophic forgetting in Vision-Language Models (VLMs) fine-tuned for autonomous driving, revealing that fine-tuning erodes pre-trained world knowledge. They introduce a new benchmark dataset (FidelityDrivingBench) to quantify this forgetting. To mitigate this, they propose Drive Expert Adapter (DEA), which adapts the model via prompt-space routing to different knowledge experts, preserving the original VLM weights.
Fine-tuning VLMs for driving erases their pre-trained world knowledge, but a novel prompt-based adapter can recover this lost generalization ability.
The integration of Vision-Language Models (VLMs) into autonomous driving promises to solve long-tail scenarios, but this paradigm faces the critical and unaddressed challenge of catastrophic forgetting. The very fine-tuning process used to adapt these models to driving-specific data simultaneously erodes their invaluable pre-trained world knowledge, creating a self-defeating paradox that undermines the core reason for their use. This paper provides the first systematic investigation into this phenomenon. We introduce a new large-scale dataset of 180K scenes, which enables the first-ever benchmark specifically designed to quantify catastrophic forgetting in autonomous driving. Our analysis reveals that existing methods suffer from significant knowledge degradation. To address this, we propose the Drive Expert Adapter (DEA), a novel framework that circumvents this trade-off by shifting adaptation from the weight space to the prompt space. DEA dynamically routes inference through different knowledge experts based on scene-specific cues, enhancing driving-task performance without corrupting the model's foundational parameters. Extensive experiments demonstrate that our approach not only achieves state-of-the-art results on driving tasks but also effectively mitigates catastrophic forgetting, preserving the essential generalization capabilities that make VLMs a transformative force for autonomous systems. Data and model are released at FidelityDrivingBench.
