Search papers, labs, and topics across Lattice.
This paper introduces TraceSQL, a traceable verification model designed to estimate the answerability of generated SQL queries without relying on ground-truth references. By leveraging 67 explicit diagnostic features related to question ambiguity, SQL structure, and intent alignment, TraceSQL provides enhanced visibility into the factors influencing its predictions. The model outperforms the GradeSQL-7B ORM baseline, achieving 66.47% F1 and 64.48% ROC-AUC, while maintaining traceability for diagnostic purposes.
TraceSQL achieves superior SQL verification accuracy while offering unprecedented transparency into the decision-making process of the model.
Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inference time in real-world deployments. This creates a critical verification problem: given only a user question, database context, and generated SQL, can a system estimate whether the generated query is likely to correctly answer the question? Recent approaches use LLMs as judge or specialized agents to inspect generated SQL, but their decisions can be difficult to trace. Outcome Reward Models (ORMs) address this by learning from execution-labeled candidate SQLs and assigning correctness scores to unseen queries, yet they still provide limited visibility into the signals behind each verification. To address this limitation, we propose TraceSQL, a lightweight and traceable verification model built on explicit diagnostic features. TraceSQL combines 67 features capturing question ambiguity, question requirements, question-schema-SQL consistency, SQL structure, and intent alignment. These signals remain available for examining which factors influence each prediction and for tracing decisions back to diagnostic evidence. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 58.26% ROC-AUC for the GradeSQL-7B ORM baseline on the same generated-SQL evaluation. Feature attribution further shows that the model relies on both semantic grounding and deterministic SQL-structure signals. These results show that SQL verification can be performed with a lightweight learned model while retaining feature-level evidence for inspecting and diagnosing its predictions.
