Search papers, labs, and topics across Lattice.
This study benchmarks human drivers and vision-language models (VLMs) in autonomous driving scenarios using dashcam footage from Lima and New York City, focusing on their performance in out-of-distribution edge cases. By employing a Visual Question Answering (VQA) framework, the research reveals that while both humans and VLMs exhibit divergent responses based on question type, geographical differences do not significantly affect their answers. The findings suggest that the challenges posed by these diverse urban environments may not be as impactful on model performance as previously anticipated, highlighting the robustness of VLMs in unfamiliar settings.
Humans and VLMs show surprising consistency in their responses across diverse urban environments, challenging assumptions about geographical influence on autonomous driving performance.
As Self-Driving Cars continue to expand internationally and use multi-modal systems such as VLMs as a cognitive backbone for their Action models; how well will these systems generalize in new settings, in particular out-of-distribution (OOD) edge-case scenarios in new geographies? In this paper, we study this open question by providing a full factorial analysis with human drivers of Lima, human drivers from New York City, and VLMs and showing them dashcam footage collected from Lima and New York City -- prompting them with a variety of questions under a Visual Question Answering (VQA) paradigm. In particular, we pick these two cities as they are highly challenging driving locations where no Self-Driving Car company currently operates in, and ask questions that span 4 categories: Factual, Ratings, Counterfactual and Reasoning. We find that Humans and VLMs diverge in their responses -- though this is modulated by the type of questions asked, and that Humans answer similarly independent of where they are from (Lima/NYC). To our surprise, we did not find a strong difference in terms of answers (Humans or VLMs) that was modulated by geography, likely due to their high out-of-distribution nature. Our dataset is available at: https://huggingface.co/datasets/Artificio/robusto-2
