Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
CArtBench, a new benchmark, evaluates VLMs on complex Chinese art tasks beyond basic recognition, using museum-sourced data aligned with expert catalogs. The benchmark includes CuratorQA (evidence-based reasoning), CatalogCaption (expert-style appreciation), Reinterpret (defensible reinterpretation), and ConnoisseurPairs (authenticity discrimination). Experiments with nine VLMs reveal limitations in evidence linking, style-to-period inference, long-form appreciation, and authenticity discrimination, even with high overall accuracy on simpler tasks.
Even state-of-the-art VLMs struggle with connoisseur-level reasoning about Chinese art, often failing at authenticity discrimination and nuanced interpretation despite appearing proficient on simpler recognition tasks.
We introduce CARTBENCH, a museum-grounded benchmark for evaluating vision-language models (VLMs) on Chinese artworks beyond short-form recognition and QA. CARTBENCH comprises four subtasks: CURATORQA for evidence-grounded recognition and reasoning, CATALOGCAPTION for structured four-section expert-style appreciation, REINTERPRET for defensible reinterpretation with expert ratings, and CONNOISSEURPAIRS for diagnostic authenticity discrimination under visually similar confounds. CARTBENCH is built by aligning image-bearing Palace Museum objects from Wikidata with authoritative catalog pages, spanning five art categories across multiple dynasties. Across nine representative VLMs, we find that high overall CURATORQA accuracy can mask sharp drops on hard evidence linking and style-to-period inference; long-form appreciation remains far from expert references; and authenticity-oriented diagnostic discrimination stays near chance, underscoring the difficulty of connoisseur-level reasoning for current models.