Search papers, labs, and topics across Lattice.
This paper introduces DEEPCHART, a benchmark designed to evaluate the fidelity of chart generation by large language models (LLMs) in data-science contexts. By analyzing 1,482 instances from diverse real-world sources, the study reveals that while LLMs can create visually appealing charts, they frequently produce data-level hallucinations due to errors in evidence extraction and reasoning. The findings indicate that merely increasing context windows is inadequate for achieving reliable chart generation; robust evidence extraction and quantitative reasoning are essential components.
Visually plausible charts generated by LLMs often mask significant data-level hallucinations, revealing a critical gap in current AI capabilities.
Faithful chart generation in real-world data-science workflows requires grounding visualizations in scattered evidence, computing chart-ready quantities, and rendering them accurately. Modern LLMs can produce visually plausible, instruction-compliant charts, yet data-level hallucinations remain difficult to detect in long, noisy, and multimodal contexts. To measure this gap, we introduce DEEPCHART, an expert-annotated benchmark of 1,482 task-conditioned chart-generation instances drawn from real-world scientific papers, financial filings, and ecosystem reports. DEEPCHART formulates chart generation as an Extract--Reason--Visualize pipeline and evaluates source-data extraction, derived-data reasoning, and chart rendering stage by stage. Experiments with state-of-the-art models show that visually plausible charts often conceal data-level hallucinations, with extraction and reasoning errors common in realistic long and multimodal settings. These findings suggest that larger context windows alone are insufficient; faithful chart generation also requires reliable evidence extraction and quantitative reasoning before rendering. Our benchmark and associated resources are available at https://github.com/tangdouer1005/DeepChart.
