Search papers, labs, and topics across Lattice.
This paper introduces SQuaD-SQL, a method that leverages knowledge distillation and synthetic data generation to enhance the performance of small language models (SLMs) on the Text-to-SQL task. By employing LLM-based synthetic data generation, parameter-efficient fine-tuning, and domain-adaptive fine-tuning, SQuaD-SQL achieves an execution accuracy of 86.9% on the WikiSQL dataset, closely rivaling larger models while significantly reducing computational demands. The findings highlight that SLMs can be effectively utilized in resource-constrained settings for efficient database interaction without sacrificing performance.
Small language models can achieve near state-of-the-art Text-to-SQL performance with just a fraction of the computational resources required by large models.
Text-to-SQL is a fundamental task in natural language processing that enables users to interact with structured databases using natural language. While large language models (LLMs) have demonstrated remarkable performance on this task, their substantial computational requirements hinder deployment in resource-constrained settings. In this paper, we introduce SQuaD-SQL (Small-Qualified and Distilled for SQL), a novel approach that empowers small language models (SLMs) to approach the performance of LLMs on the Text-to-SQL task while significantly improving efficiency through knowledge distillation and synthetic data generation. Our method comprises three key components: (1) LLM-based synthetic data generation, where structured knowledge is extracted from LLMs via carefully designed prompting strategies; (2) parameter-efficient fine-tuning, enabling full model training on a single consumer-grade GPU; and (3) domain-adaptive fine-tuning, where domain-specific synthetic data further enhances performance in targeted domains. Experiments on the WikiSQL dataset demonstrate that SQuaD-SQL achieves an execution accuracy of 86.9% on the test set, approaching the performance of LLMs while offering faster inference and lower memory usage. These results suggest that, with proper training strategies, SLMs can serve as practical and efficient alternatives for Text-to-SQL applications in resource-limited environments.
