Search papers, labs, and topics across Lattice.
This paper introduces StatABench, a comprehensive benchmark designed to evaluate the statistical analysis capabilities of large language models (LLMs) through two components: Stat-Closed and Stat-Open. The evaluation of various LLMs, including GPT-5.1, reveals that even the most advanced models struggle to achieve high accuracy, with scores of only 68.6% on Stat-Closed and 61.86% on Stat-Open tasks. These findings underscore significant limitations in LLMs' ability to perform reliable statistical analysis, particularly in areas requiring tool-grounded reasoning and methodological decision-making.
LLMs struggle with statistical analysis, achieving only 68.6% accuracy on a new benchmark designed to rigorously test their capabilities.
Statistical analysis is a broad, complex field requiring both domain knowledge and tool proficiency. While prior work has evaluated large language models (LLMs) in this domain, existing benchmarks remain limited in scope and format. To bridge this gap, we introduce StatABench (Statistical AnalysisBenchmark), a benchmark designed to systematically assess LLMs' statistical analysis capabilities. StatABench comprises two complementary components: Stat-Closed, containing 404 questions across 18 statistical topics in multiple formats (multiple-choice, fill-in-the-blank, decision-making, and practical application), and Stat-Open, featuring 30 complex open-ended modeling tasks adapted from professional competitions. We evaluate diverse LLMs using the LangChain MCP framework and multiple data science agents, and assess Stat-Open solutions via a validated LLM-as-Judge protocol. Experiments show that even GPT-5.1 achieves only 68.6% on Stat-Closed, while the best open-source model reaches 60.6%. On Stat-Open, the top agent framework scores 61.86 on average. These results reveal the gap between current LLMs and reliable statistical analysis, highlighting persistent challenges in tool-grounded reasoning, methodological decision-making, and end-to-end statistical modeling.
DAMO