Search papers, labs, and topics across Lattice.
This paper introduces HakushoBench, a Japanese benchmark for visual question answering (VQA) focused on charts and tables derived from governmental white papers. By leveraging a diverse set of 2,053 images across various formats, the benchmark aims to evaluate the deep understanding of visual data rather than just local cues, addressing the scarcity of non-English VQA resources. Experimental results reveal that even the best open-weight models struggle with this benchmark, achieving only 58.6% accuracy, indicating significant challenges in complex chart and table comprehension across languages.
Open-weight models lag significantly behind proprietary counterparts on a new Japanese VQA benchmark, highlighting critical gaps in chart and table understanding.
Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchmarks have advanced rapidly, non-English counterparts remain scarce, leaving it unclear whether this progress generalizes across languages. A key obstacle is the difficulty of collecting realistic and diverse non-English chart and table images at scale. To address this, we leverage governmental white papers as a scalable source for benchmark construction beyond English, as they contain naturally occurring charts and tables across diverse formats and domains and are freely accessible in many countries. As a first instantiation, we introduce HakushoBench, a challenging Japanese chart and table VQA benchmark built from 33 governmental white papers. HakushoBench contains 2,053 images spanning over 10 image types, with manually annotated QA pairs, designed to assess deep and holistic understanding of charts and tables, rather than local visual cues alone. Experiments across a broad range of VLMs demonstrate that HakushoBench remains challenging for open-weight models: the best open-weight model achieves only 58.6% accuracy, and a 34.9-point gap between open-weight and proprietary models highlights substantial room for improvement in complex chart and table understanding. We release our dataset and code.
