Search papers, labs, and topics across Lattice.
This paper introduces LakeQuest, a benchmark consisting of 9,846 question-answer pairs specifically designed to evaluate grounded question answering (QA) systems over heterogeneous data lakes. By focusing on real-world scenarios across three domains鈥擜I/ML metadata, retail banking, and multimodal biomedical drug information鈥擫akeQuest assesses the end-to-end performance of QA systems in retrieving and synthesizing information from weakly structured collections. The findings indicate that even high-quality retrieval does not ensure accurate reasoning, exposing significant weaknesses in current QA approaches, particularly in relation chaining and cross-modal synthesis.
High-quality retrieval fails to guarantee correct reasoning in real-world QA systems, revealing critical vulnerabilities in their performance.
While modern question answering (QA) systems excel on clean, schema-aligned corpora, real-world knowledge is rarely so neatly packaged. Answering questions over enterprise and scientific data lakes requires systems to navigate heterogeneous, weakly structured collections of tables, passages, and linked metadata. Current benchmarks abstract away this noisy discovery process, failing to evaluate end-to-end performance. To bridge this gap, we introduce LakeQuest, a human-validated benchmark of 9,846 QA pairs designed to evaluate the end-to-end retrieve-and-synthesize pipeline over realistic data lakes. LakeQuest spans three diverse domains (AI/ML metadata, retail banking, and multimodal biomedical drug information) and pairs every question with exact, modality-aware evidence pointers. By isolating source discovery from cross-modal synthesis, LakeQuest exposes critical failure modes in modern QA systems. Our baseline evaluations, including standard Retrieval-Augmented Generation (RAG) and agentic tool-use methods, reveal that high-quality retrieval does not guarantee correct reasoning. Systems consistently struggle with relation chaining in metadata graphs, policy grounding in bank ledgers, and joint tabular QA in biomedical contexts, highlighting the need for robust discovery and faithful cross-file composition mechanisms in future agentic QA systems.
