Search papers, labs, and topics across Lattice.
This paper introduces ACT-Eval, a novel evaluation framework that dissects chess commentary into atomic claims, allowing for the assessment of factual correctness and move quality through engine-supported tools and expert-annotated references. The study evaluates leading language models, revealing that GPT-5.4 produces incorrect sub-claims 22.0% of the time, with smaller models exceeding 40% error rates, despite improvements from tool augmentation. The findings highlight that while factual accuracy can be enhanced, the models still struggle with capturing expert-level strategic and tactical insights, indicating a gap in their educational utility.
Factual hallucinations in LLM chess commentary are alarmingly high, with smaller models making over 40% incorrect claims, even after tool augmentation.
Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that make such expertise educationally useful to experts and non-experts alike. Large language models could, in principle, bridge this gap, but they frequently hallucinate due to limited domain-specific knowledge, and standard reference-based or LLM-as-a-judge frameworks cannot reliably detect these errors. In this work, we present ACT-Eval, an evaluation framework that decomposes chess commentary into atomic claims and routes them to engine-supported tools and expert-annotated gold references to assess factual correctness, conceptual coverage, and move-quality judgment. We release a benchmark of 325 position--move pairs spanning pedagogical, tournament, and critical positions, including 125 positions with expert-verified gold atoms and a five-class error taxonomy. Evaluating leading proprietary and open-weight models, we find that factual hallucinations remain pervasive in chess commentary: GPT-5.4 without tools produces incorrect sub-claims 22.0% of the time, while smaller open-weight models exceed 40%. Although tool augmentation substantially improves factual correctness and move-quality assessment, coverage of expert strategic and tactical ideas remains limited across all models. Human calibration shows that ACT-Eval's factual judgments fall within the observed range of inter-human agreement, while its coverage scores correlate strongly with human assessments of strategic completeness.
UW