Search papers, labs, and topics across Lattice.
The paper introduces VENUSS, a framework for systematically evaluating VLMs on sequential driving scenes, revealing performance gaps and sensitivities to input configurations. They benchmarked 25+ VLMs on 2,600+ extracted driving video sequences, finding a top accuracy of only 57%, significantly below human performance (65%). The analysis highlights VLM limitations in understanding vehicle dynamics and temporal relationships within driving scenarios, despite proficiency in static object detection.
Even state-of-the-art VLMs stumble when reasoning about sequential driving scenes, achieving only 57% accuracy compared to human-level 65%, exposing critical gaps in understanding vehicle dynamics and temporal relations.
Vision-Language Models (VLMs) are increasingly proposed for autonomous driving tasks, yet their performance on sequential driving scenes remains poorly characterized, particularly regarding how input configurations affect their capabilities. We introduce VENUSS (VLM Evaluation oN Understanding Sequential Scenes), a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes, establishing baselines for future research. Building upon existing datasets, VENUSS extracts temporal sequences from driving videos, and generates structured evaluations across custom categories. By comparing 25+ existing VLMs across 2,600+ scenarios, we reveal how even top models achieve only 57% accuracy, not matching human performance in similar constraints (65%) and exposing significant capability gaps. Our analysis shows that VLMs excel with static object detection but struggle with understanding the vehicle dynamics and temporal relations. VENUSS offers the first systematic sensitivity analysis of VLMs focused on how input image configurations - resolution, frame count, temporal intervals, spatial layouts, and presentation modes - affect performance on sequential driving scenes. Supplementary material available at https://V3NU55.github.io
