Search papers, labs, and topics across Lattice.
The paper introduces OneDrive, a unified autonomous driving framework built upon a pretrained Vision-Language Model (VLM) that handles heterogeneous decoding behaviors (language generation, object detection, trajectory regression) within a single transformer decoder. It leverages the transferability of pretrained VLM attention by organizing visual and structured query tokens within a causal decoder, allowing structured queries to condition on visual context. Experiments on autonomous driving benchmarks demonstrate state-of-the-art performance and competitive closed-loop results, along with multi-modal generation capability and reduced latency.
Ditch the fragmented architectures: OneDrive unifies autonomous driving tasks within a single VLM decoder, achieving state-of-the-art performance while slashing latency.
Vision-Language Models(VLMs) excel at autoregressive text generation, yet end-to-end autonomous driving requires multi-task learning with structured outputs and heterogeneous decoding behaviors, such as autoregressive language generation, parallel object detection and trajectory regression. To accommodate these differences, existing systems typically introduce separate or cascaded decoders, resulting in architectural fragmentation and limited backbone reuse. In this work, we present a unified autonomous driving framework built upon a pretrained VLM, where heterogeneous decoding behaviors are reconciled within a single transformer decoder. We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling. By organizing visual and structured query tokens within a single causal decoder, structured queries can naturally condition on visual context through the original attention mechanism. Textual and structured outputs share a common attention backbone, enabling stable joint optimization across heterogeneous tasks. Trajectory planning is realized within the same causal LLM decoder by introducing structured trajectory queries. This unified formulation enables planning to share the pretrained attention backbone with images and perception tokens. Extensive experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance, including 0.28 L2 and 0.18 collision rate on nuScenes open-loop evaluation and competitive results (86.8 PDMS) on NAVSIM closed-loop evaluation. The full model preserves multi-modal generation capability, while an efficient inference mode achieves approximately 40% lower latency. Code and models are available at https://github.com/Z1zyw/OneDrive
