Search papers, labs, and topics across Lattice.
This paper investigates the trade-off between model size and inference compute in the context of grammar-constrained text-to-SQL tasks using the Qwen2.5-Instruct model family. The authors find that while both beam search and sample+vote methods enhance accuracy, increasing model size generally yields better results than merely increasing inference compute, particularly for smaller models. Notably, beam search outperforms sample+vote under matched inference budgets, highlighting a divergence from previously established unconstrained trade-off behaviors.
Increasing model size trumps inference compute for grammar-constrained text-to-SQL tasks, with beam search proving superior to sample+vote under matched budgets.
One common trade-off in the use of large language models involves reducing the size of the model while increasing the amount of computation at inference time, for example by using a wider beam search. In this paper, we examine the constrained case of this"model size vs. inference compute"trade-off, in which the model outputs are constrained by a strict grammar at inference time. Our results demonstrate that the constrained trade-off behaves differently from the unconstrained trade-off. We investigate the task of converting a prose query into an equivalent SQL query (text-to-SQL). Performance is evaluated on the Spider text-to-SQL benchmark, using the Qwen2.5-Instruct model family ranging in size from 0.5B to 7B parameters, all at 4-bit precision. We experiment with two approaches to varying inference compute: (i) beam search with a variable number of beams; and (ii) sample+vote, i.e., sampling several constrained outputs and then voting on their execution results, where the number of samples is varied. On the 1034-example development set, we find that: (a) both beam search and sample+vote improve accuracy, especially on smaller model sizes; (b) the"model size vs.\ inference compute"trade-off is not advantageous in this experiment, because moving to a larger model size typically results in higher accuracy than increasing inference compute on the same model size; (c) beam search outperforms sample+vote at a matched inference budget. This latter result is of particular interest since it contrasts with the findings of the unconstrained trade-off.
