Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Chartography, a new benchmark consisting of 100 tasks designed to evaluate professional chart understanding across various domains, addressing the limitations of existing benchmarks that primarily focus on simpler chart types and shorter reasoning chains. The evaluation of 30 advanced model configurations reveals that even the best-performing model achieves only 45.0% mean pass@1, indicating significant shortcomings in visual perception and reasoning capabilities when interpreting complex charts. The findings highlight critical areas where current models struggle, such as misreading values and failing to recognize nuanced features, underscoring the need for improved methodologies in chart comprehension tasks.
Despite advancements in AI, models still struggle with professional chart interpretation, achieving only 45% accuracy on a new benchmark designed for real-world applications.
Professionals across medicine, engineering, finance, manufacturing, and the sciences often make consequential decisions from charts. Existing chart benchmarks do not sufficiently measure this ability: they are dominated by bar, line, and pie formats, rely on shorter reasoning chains, and are nearing saturation, with frontier models already scoring 80-90%. We introduce Chartography, a benchmark of 100 tasks that pair charts drawn from professional practice, in domain-specific formats that standard chart benchmarks rarely include, with questions written by professionals who read these charts for a living and independently verified by three additional experts. In an evaluation of 30 frontier-model configurations (20 scored trials per task), the best configuration reaches only 45.0% mean pass@1; the remainder span 9.0-39.5%. Failures concentrate in visual perception: models can miss nuanced features, misread values along sparsely labeled axes, mishandle projected 3D geometry, and violate domain conventions encoded in the chart. We release all tasks, images, provenance metadata, and evaluation code.