Search papers, labs, and topics across Lattice.
This paper introduces VOLA, a novel approach for enhancing open-world driving by predicting semantic attributes that dictate how vehicles should interact with various objects, rather than merely identifying them. By employing a framework that focuses on dense action-relevant attributes鈥攕pecifically drivability and vulnerability鈥擵OLA generates high-resolution attribute maps from Qwen3.5 image-token hidden states without relying on autoregressive text generation or external mask models. The method demonstrates superior transferability to real-world scenarios, achieving a mean vulnerability-rank recall of 69.4%, significantly outperforming traditional vision-only segmenters and prompted VLM segmenters.
VOLA achieves a 69.4% mean vulnerability-rank recall, outperforming existing models by effectively translating visual cues into actionable driving attributes for open-world scenarios.
Driving in the real world is open-world: a car may encounter a fallen mattress, a deer, or other objects outside its training data. Naming them is not enough. The system must know how to treat each region: can it drive over it, and how severe would a collision be? We therefore shift scene perception from category labels to dense action-relevant attributes, where each pixel is labeled by how it should affect motion rather than by object name. We instantiate this general formulation with two ordered attributes: 7-rank drivability and 5-rank vulnerability. We read Qwen3.5 image-token hidden states directly as a spatial semantic representation. A lightweight boundary-aware decoder then turns this coarse token grid into sharp full-resolution attribute maps. The whole process requires neither autoregressive text generation nor an external mask model such as SAM. We train on dense attribute labels built in CARLA and test transfer to real scenes and to novel obstacles never seen in training. We compare with vision-only segmenters trained on the same attributes and prompted VLM segmenters. Our model matches strong vision-only segmenters on familiar categories and improves transfer to real open-world anomalies, reaching 69.4% mean vulnerability-rank recall versus 57.1% for the best vision-only baseline and 53.9% for the best prompted VLM baseline. These results show that VLM image tokens provide useful semantic cues for transferring driving attributes to objects outside the training vocabulary.
