Search papers, labs, and topics across Lattice.
This paper introduces DataGovBench, a novel benchmark designed to evaluate Large Language Models (LLMs) against the complexities of real-world data analysis, specifically focusing on large multi-tabular datasets and exploratory insight generation. Through two main tasks鈥擳able QA and Table Insight鈥攖he authors assess the models' capabilities in answering complex questions and generating expert-level findings. The results reveal significant performance gaps, indicating that current LLMs are not yet equipped to meet the demands of practical data analytics, highlighting the need for further advancements in this area.
Current LLMs struggle with real-world data complexities, revealing critical performance gaps in data analysis tasks that could hinder their practical applications.
Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.
