Search papers, labs, and topics across Lattice.
This paper introduces TrustDABench, a benchmark designed to evaluate the reliability and robustness of large language models (LLMs) in structured data analysis, specifically focusing on their ability to trace valid evidence paths and maintain analysis integrity across different data representations. The evaluation of eight LLMs revealed significant shortcomings, with the highest reliability score at only 24.21% and robustness at 9.10%, indicating a systematic failure in recognizing conflicting evidence and adhering to valid analytical paths. These results highlight the urgent need for improved mechanisms in LLMs to enhance their reliability when analyzing structured data.
LLMs struggle with structured data analysis, achieving only 24.21% reliability and 9.10% robustness, revealing critical gaps in their reasoning capabilities.
LLMs are increasingly used to analyze spreadsheets, CSV files, and other structured data, but producing a correct-looking answer is not the same as producing a trustworthy analysis. A trustworthy result should be supported by a valid path from the user question to the relevant data evidence. This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms. We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness. Starting from the evidence-path view, we derive 19 perturbation operators and instantiate them through an Agentic-LLM-based generation framework. TrustDABench contains 2,340 human-verified perturbed instances, and we evaluate eight representative LLMs. The results show substantial headroom: the best reliability result is only 24.21% average MRS, achieved by GPT-5.5, while the best robustness result still has 9.10% average ASR, achieved by Claude-Sonnet-5. The failures are systematic: models rarely detect conflicting evidence, often continue along executable but unsupported analysis paths, and remain sensitive to perturbations that change observation boundaries or cross-table relations. These findings suggest that stronger evidence-boundary recognition and representation-invariant reasoning are still needed for reliable structured-data analysis.
