Search papers, labs, and topics across Lattice.
This paper introduces SPENCE, a syntactic probing framework designed to detect and quantify contamination in NL2SQL benchmarks, addressing concerns over inflated accuracy in LLM performance due to training data leakage. By systematically generating syntactic variants of test queries across four major NL2SQL datasets, the authors evaluate the robustness of various high-capacity LLMs and measure execution accuracy changes with increasing syntactic divergence. The results reveal a significant temporal gradient in contamination levels, with older benchmarks like Spider showing high susceptibility to training leakage, while newer datasets like BIRD remain largely unaffected, underscoring the need for rigorous evaluation methods in benchmark integrity.
Contamination in NL2SQL benchmarks can inflate LLM accuracy, but SPENCE reveals that newer datasets are largely free from this bias.
Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. We introduce SPENCE (Syntactic Probing and Evaluation of NL2SQL Contamination Effects), a controlled syntactic probing framework for detecting and quantifying such contamination. SPENCE systematically generates syntactic variants of test queries for four widely used NL2SQL datasets-Spider, SParC, CoSQL, and the newer BIRD benchmark. We use SPENCE to evaluate multiple high-capacity LLMs under execution-based scoring. For each model, we measure changes in execution accuracy across increasing levels of syntactic divergence and quantify rank sensitivity using Kendall's tau with bootstrap confidence intervals. By aligning these robustness trends with benchmark release dates, we observe a clear temporal gradient: older benchmarks such as Spider exhibit the strongest negative values and thus the highest likelihood of training leakage, whereas the more recent BIRD dataset shows minimal sensitivity and appears largely uncontaminated. Together, these findings highlight the importance of temporally contextualized, syntactic-probing evaluation for trustworthy NL2SQL benchmarking.
