Search papers, labs, and topics across Lattice.
This paper audits existing Knowledge-Based Visual Question Answering (KB-VQA) benchmarks, revealing critical flaws such as missing or contradicted answers and underspecified questions that undermine the reliability of answer accuracy as a metric. The authors introduce a two-part solution: a principled audit-and-repair protocol to enhance answer derivability and question clarity, and a controlled multi-entity augmentation protocol to introduce necessary visual ambiguity. Their findings indicate that re-evaluating models under these corrected and augmented conditions significantly alters performance trends, suggesting a need for more robust evaluation protocols in KB-VQA.
Existing KB-VQA benchmarks mislead model evaluation, with flawed assumptions leading to overestimated reasoning capabilities in Visual Language Models.
Knowledge-Based Visual Question Answering (KB-VQA) aims to evaluate whether Visual Language Models (VLMs) can retrieve, ground, and reason over external structured knowledge beyond visual evidence. In practice, answer accuracy is widely adopted as the primary evaluation metric, implicitly treating correctness as a proxy for knowledge-grounded reasoning. However, for existing KB-VQA benchmarks, this proxy relies on critical assumptions that are often overlooked and rendered unreliable by benchmark issues: annotated answer must be derivable from the associated knowledge base, question must be well-posed with sufficient constraints, and visual setting must meaningfully require grounded disambiguation. In this work, we show that these assumptions are systematically violated in existing KB-VQA benchmarks. Our audit reveals substantial instances with missing or contradicted answers and underspecified questions that render accuracy a misleading metric. Furthermore, we find that existing datasets rely on visually trivial, single-entity scenes that bypass the need for sophisticated visual-to-knowledge mapping. We demonstrate that even with controlled architectures, these flaws lead to distorted model rankings and overestimations of reasoning capabilities. To address this, we introduce (1) a principled audit-and-repair protocol that restores answer derivability and question clarity, and (2) a controlled multi-entity augmentation protocol that introduces visual ambiguity to challenge initial retrieval and grounded reasoning. Re-evaluation under corrected and augmented settings yields markedly different performance trends. Our findings call for rethinking evaluation protocols and designing more interaction-aware KB-VQA benchmarks that prioritize verifiable reasoning over simple matching.
