Search papers, labs, and topics across Lattice.
This paper presents EduArt, a benchmark designed to evaluate art history knowledge and visual reasoning in large language models (LLMs) using 871 human-authored questions from secondary and advanced placement art history curricula. The evaluation of twelve models revealed that while multiple-choice formats led to high accuracy, models performed significantly worse on open-ended tasks and error identification, indicating a disparity between recognition and application of art-historical knowledge. These findings underscore the limitations of single-format assessments, suggesting that they may overestimate LLM capabilities in nuanced academic contexts.
Models that excel in multiple-choice tasks can falter dramatically in open-ended and error identification formats, revealing a critical gap in art-historical reasoning capabilities.
Large language models now score near ceiling on general benchmarks, but these aggregate measures reveal little about how models behave within single disciplines. Existing art-focused evaluations rely on synthetic questions and rarely report item-level properties. This paper introduces EduArt, an educational-level benchmark for art-historical knowledge and visual reasoning in multimodal LLMs. EduArt comprises 871 human-authored questions from Italian secondary-school exercises and US Advanced Placement Art History exams, spanning two languages and seven formats from multiple choice to in-text word placement and error identification. Twelve models from six provider families were evaluated under a default answer-only condition and a motivation condition requiring written justification, and characterized using Classical Test Theory and a logistic regression isolating the effects of format, language, image presence, and model. The benchmark showed strong psychometric properties (mean discrimination 0.514, 82.3 percent good discriminators), while multiple-choice accuracy saturated near ceiling for six models, showing recognition formats alone cannot distinguish frontier models. Format was a strong independent predictor of accuracy: models exceeding 94 percent on multiple choice fell to 23.9 percent on open completion (Claude Opus 4.6) and 6.2 percent on error identification (Claude Sonnet 4.6). The motivation condition changed accuracy in a predominantly negative, family-dependent direction. These dissociations indicate that art-historical knowledge and the ability to deploy it are distinct capabilities, and that single-format benchmarks overestimate what models can reliably do. Mapping this capability profile is a precondition for responsible use of multimodal LLMs in art-historical scholarship, where tasks demand producing and manipulating content rather than selecting from fixed options.
