Search papers, labs, and topics across Lattice.
The paper introduces TPS-Drive, a framework that addresses the challenge of bridging semantic reasoning and precise 3D spatial forecasting in Vision-Language Model (VLM)-based autonomous driving. It uses a task-guided vector quantization mechanism within an Agent-Centric Tokenizer, supervised by a frozen 3D detection head, to purify spatial representations by reallocating codebook capacity from static backgrounds to dynamic agents. TPS-Drive achieves state-of-the-art performance on nuScenes, NAVSIMv1, and NAVSIMv2 benchmarks by decoupling reasoning into scene understanding, future forecasting, and action generation, optimized through a progressive three-stage training paradigm.
By intelligently filtering out irrelevant visual noise, TPS-Drive lets VLMs focus on what *really* matters for safe self-driving: dynamic agents, not static scenery.
Vision-Language Models (VLMs) provide a promising foundation for autonomous driving planning, yet bridging semantic reasoning and precise 3D spatial forecasting remains a critical challenge. Existing representation strategies generally follow two paths: text-aligned methods flatten continuous spatial states into symbols, which compromises geometric structure and induces"spatial hallucinations"; dense visual methods preserve spatial topology but overwhelm standard tokenizers with redundant background textures, leading to"representation interference". To address these limitations, we introduce TPS-Drive, a novel framework centered on Task-Guided Representation Purification that empowers VLMs to Think in Purified Space. At its core, an Agent-Centric Tokenizer utilizes a task-guided vector quantization mechanism supervised by a frozen 3D detection head, which explicitly reallocates limited codebook capacity from pervasive static backgrounds to critical dynamic agents and effectively isolates spatial redundancy. Leveraging this purified spatial vocabulary, TPS-Drive employs a decoupled reasoning pipeline that sequentially performs scene understanding, future forecasting, and action generation. The framework is optimized via a progressive three-stage training paradigm, culminating in reward-driven refinement that surpasses pure imitation learning. Extensive experiments validate our approach: TPS-Drive achieves accurate agent spatial state forecasting and reduces collision rates in open-loop nuScenes evaluations, while establishing new safety records on the rigorous closed-loop NAVSIMv1 and NAVSIMv2 benchmarks.
