Search papers, labs, and topics across Lattice.
This paper investigates the ability of Large Language Models (LLMs) to understand and enforce data visualization rules, comparing their performance to constraint-based systems like Draco. The authors created a dataset of 2,000 Vega-Lite specifications annotated with rule violations, derived from a subset of Draco's constraints translated into natural language. Results show that while LLMs achieve high prompt adherence and detect common violations reasonably well, their performance significantly drops for subtle perceptual rules and outputs generated from technical ASP formulations, indicating limitations compared to symbolic solvers.
LLMs can reliably detect common data visualization violations, but struggle with subtle perceptual rules, revealing a gap between their general reasoning abilities and specialized visual design knowledge.
Data visualization rules-derived from decades of research in design and perception-ensure trustworthy chart communication. While prior work has shown that large language models (LLMs) can generate charts or flag misleading figures, it remains unclear whether they can reason about and enforce visualization rules directly. Constraint-based systems such as Draco encode these rules as logical constraints for precise automated checks, but maintaining symbolic encodings requires expert effort, motivating the use of LLMs as flexible rule validators. In this paper, we present the first systematic evaluation of LLMs against visualization rules using hard-verification ground truth derived from Answer Set Programming (ASP). We translated a subset of Draco's constraints into natural-language statements and generated a controlled dataset of 2,000 Vega-Lite specifications annotated with explicit rule violations. LLMs were evaluated on both accuracy in detecting violations and prompt adherence, which measures whether outputs follow the required structured format. Results show that frontier models achieve high adherence (Gemma 3 4B / 27B: 100%, GPT-oss 20B: 98%) and reliably detect common violations (F1 up to 0.82),yet performance drops for subtler perceptual rules (F1 < 0.15 for some categories) and for outputs generated from technical ASP formulations.Translating constraints into natural language improved performance by up to 150% for smaller models. These findings demonstrate the potential of LLMs as flexible, language-driven validators while highlighting their current limitations compared to symbolic solvers.
