Search papers, labs, and topics across Lattice.
This paper introduces Spider 2.0-AIFunc, a benchmark designed to evaluate the generation of AI-native SQL queries, addressing a gap in existing text-to-SQL benchmarks that only focus on conventional SQL. By employing an agent-based pipeline, the authors transform traditional SQL tasks into AI-native SQL queries, refining natural language instructions to enhance clarity and reduce ambiguity. Evaluation of ten state-of-the-art language models reveals that proprietary models achieve execution accuracy between 67-70%, while the best open-source model reaches only 58.1%, highlighting significant challenges in predicate specification and schema grounding in AI-native contexts.
AI-native SQL queries expose critical gaps in model performance, with top proprietary models still struggling to achieve 70% execution accuracy.
Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .
