Search papers, labs, and topics across Lattice.
This paper introduces RoboVista, a benchmark designed to evaluate Vision-Language Models (VLMs) across diverse robotic applications, addressing the limitations of traditional benchmarks that lack modularity and real-world relevance. By employing the Robot Question Answering (RQA) framework, the authors curated a dataset of 474 Visual Question Answering instances that encompass 39 unique task types across various domains such as agriculture and autonomous driving. Experimental results reveal significant performance gaps in state-of-the-art VLMs, with physical robot tests indicating a strong correlation between RoboVista scores and actual task execution success.
State-of-the-art Vision-Language Models fall short in real-world robotic applications, revealing critical gaps in their reasoning capabilities.
Diverse applications for robotics, such as industry and agriculture, require robots to operate across various embodiments, changing visual conditions, and complex planning. Vision-Language Models (VLMs) offer a promising foundation for general-purpose and interpretable robotic reasoning. Aligning VLMs with diverse robot applications requires a modular understanding of the individual decision components that underlie robotic behavior. Capturing such structure is challenging for conventional robot benchmarks that are primarily based on teleoperated, end-to-end datasets. We propose Robot Question Answering (RQA), a modular evaluation framework and RoboVista, a benchmark curated from real robotic systems, research papers, and expert annotations. RoboVista contains 474 Visual Question Answering (VQA) instances with human annotated reasoning and covers 39 unique task types in agricultural, industrial, domestic, surgical robotics, autonomous driving, and open robot datasets. Experiments on RoboVista show that state-of-the-art VLMs exhibit substantial gaps. Physical robot experiments suggest strong correlation between RoboVista performance and real-world task execution.
BAIR
UW