Search papers, labs, and topics across Lattice.
This paper introduces ChartAnno, a benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to generate annotations for existing charts. The study reveals that while proprietary models outperform open-source counterparts, the latter are closing the gap, particularly when given more specific instructions. Key findings indicate that inferring abstract intent poses significant challenges for MLLMs, and the inclusion of chart images yields limited improvements in annotation quality.
Proprietary MLLMs excel in chart annotation, but open-source models are rapidly catching up, especially with clearer instructions.
Multimodal large language models (MLLMs) have made significant progress in chart understanding, generation, and editing, but their ability to annotate existing charts remains underexplored. Annotating charts is a common yet challenging communicative task, requiring models to infer intended messages, interpret chart semantics, and place appropriate textual or graphical elements. To address this gap, we introduce ChartAnno, a benchmark for evaluating MLLMs on chart annotation generation. It contains 1,200 real-world charts with paired code and annotation instructions across three levels of instruction specificity. We evaluate 10 representative MLLMs under two primary input settings: (1) chart code alone and (2) both chart code and chart image, and further include a chart image-only ablation study. Results show that proprietary models remain stronger overall, although large-scale open-source models narrow the gap. More specific instructions improve annotation quality, while inferring abstract intent remains most difficult for current MLLMs. Providing chart images brings limited overall gains, with improvements mainly appearing in design-related metrics. These findings highlight chart annotation generation as a challenging task requiring semantic grounding and effective annotation design. Code and data will be released in a future version.
