Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Current Text-to-SQL benchmarks rely on brittle binary execution accuracy and lack the schema complexity of enterprise environments, masking severe model failure modes. To fix this, SQLMorph combines automated query mutations—Join Query Expansion (JQE) for structural multi-table complexity and Textual Query Augmentation (TQA) for linguistic perturbations—with continuous Execution Precision and Recall metrics. Stress-testing state-of-the-art models reveals that simple linguistic variations like abbreviations degrade accuracy by up to 17%, while execution-level F1 scoring exposes critical over- and under-retrieval patterns invisible to binary metrics.
Standard binary evaluations mask critical retrieval failures in Text-to-SQL agents, where minor conversational abbreviations slash model accuracy by 17% and increasing join depth triggers undetected multi-table degradation.
Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven by large language models (LLMs), evaluation remains a major bottleneck: public benchmarks fail to capture the complexity of enterprise schema, while building private evaluation sets is costly and nondeterministic, making evaluation results difficult to reproduce. To address this issue, we present SQLMorph, a framework for Text-to-SQL evaluation via query mutation. SQLMorph introduces two techniques to automatically generate and expand evaluation sets: Join Query Expansion (JQE), which systematically increases structural complexity through valid join additions, and Textual Query Augmentation (TQA), which generates controlled natural language perturbations to assess robustness to linguistic variation. JQE and TQA create targeted choke points to challenge specific system components. When applied to state-of-the-art systems, JQE increases query coverage and reveals accuracy degradation as the number of joins grows. Meanwhile, TQA shows that linguistic brittleness induced by heavy abbreviation can reduce accuracy by up to 17%. Beyond evaluation sets, SQLMorph introduces a family of execution-level metrics that address the limitations of current binary measures, such as Execution Accuracy. We define Execution Precision (EXP) and Execution Recall (EXR) to quantify the fraction of correct and recovered results, respectively, and combine them via F1 for unified scoring. Our experiments show that these relaxed metrics enable fine-grained analysis of over- and under-prediction, revealing differences across systems that binary metrics obscure. Together, SQLMorph's query mutation and fine-grained metrics support debugging and better align Text-to-SQL evaluation practices with real-world deployments.