Search papers, labs, and topics across Lattice.
This paper introduces a novel framework that leverages Vision-Language Models (VLMs) to identify and rank occluded agents critical to the trajectory of autonomous vehicles, addressing the limitations of existing methods that treat occlusions uniformly. By employing Planning KL-divergence (PKL) as an information-theoretic metric, the authors systematically assess the impact of hidden agents on vehicle planning. The results reveal that fine-tuning VLMs on PKL-guided data leads to significant performance enhancements, with smaller models outperforming larger zero-shot models by approximately 30%.
Smaller, fine-tuned models can outperform larger counterparts by 30% when trained to focus on planning-critical occlusions in autonomous driving scenarios.
Autonomous vehicles must safely navigate complex environments where planning-critical agents may be hidden from view. Current approaches often treat all occlusions with uniform conservatism, yielding needlessly defensive driving, or they infer hidden spaces without estimating the impact on the planner. This work bridges the critical gap between perception and planning by enabling Vision-Language Models (VLMs) to identify and reason about the specific hidden agents that are most critical to the ego-vehicle's trajectory. We introduce a novel framework that uses Planning KL-divergence (PKL), an information-theoretic metric, to systematically identify and rank occluded agents based on their impact on the ego vehicle's plan. Using this planning-aware ranking, we employ an expert VLM (GPT-5) to generate rich, structured annotations that capture the visual evidence and reasoning required for this task. We apply this framework to the nuScenes dataset to create a new benchmark focused on high-impact scenarios. We conduct comprehensive experiments on a wide range of general-purpose and domain-adapted VLMs, demonstrating that fine-tuning on our PKL-guided data yields dramatic performance improvements across all models. Notably, our results show that smaller, fine-tuned models significantly outperform their much larger zero-shot counterparts, and that our PKL-guided data selection strategy improves performance by approximately 30\% over random sampling. Our work presents the first systematic approach for training VLMs to focus on planning-critical occlusions, enabling more semantically grounded and efficient risk assessment in autonomous driving.
