Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces AnnoBench, a structured benchmark designed to evaluate the quality of visualization annotations by addressing the complex interplay of visual, semantic, and stylistic constraints. By pairing professional visualizations with diverse annotation tasks and employing a VLM-as-a-judge approach, the authors assess how various factors influence annotation quality through systematic experiments. The findings highlight critical insights into the conditions that affect annotation effectiveness, laying groundwork for improved automation in visualization tasks.
AnnoBench reveals that the right combination of input representation and prompt specificity can significantly enhance the quality of visualization annotations, challenging existing assumptions in the field.
Annotation is among the most demanding visualization tasks to automate, as it simultaneously requires correctly navigating visual, semantic, and stylistic constraints. Failure to meet any of these conditions severely undermines the utility of an annotation, rendering it challenging to read, inaccurate, or visually discordant. Despite a growing body of annotation tools and automations, no existing benchmark or evaluation framework tests whether these conditions are met because of their scope and annotation not being the focus of their studies. We introduce AnnoBench, a benchmark for visualization annotation that materializes the inherent challenges of this domain in a structured and testable manner. AnnoBench pairs visualizations from professional data journalism and visualization galleries with annotation tasks, spanning four representation formats, five chart description conditions, and two prompt specification levels. The benchmark is executed via VLM-as-a-judge, using models aligned with manual human assessment. We evaluate the benchmark via four one-factor-at-a-time experiments, exploring the effects of input representation, semantic context, and prompt specificity, and model selection on annotation quality. This work provides a foundation for advancing annotation automation, tooling, and visualization-generation pipelines.