Search papers, labs, and topics across Lattice.
This study disentangles two synthetic data scaling methods: Source Expansion (SE) and Fixed-Source Synthesis (FSS), by isolating FSS through a controlled approach that maintains a fixed seed-question pool while varying the response budget under Rejection Sampling (RS). The researchers derive a rectified scaling law for FSS, demonstrating that while SE can outperform FSS at large budgets, FSS remains a robust method for comparing synthesis protocols at matched total-sample budgets. Ultimately, the findings reveal that simply generating more responses does not necessarily enhance performance, particularly within the FSS framework, where traditional RS remains superior.
Scaling synthetic data isn't just about generating more; fixed-source synthesis reveals surprising limits to performance gains.
Synthetic data can be scaled along two routes: Source Expansion (SE), which enlarges the source by adding seed materials or generators, and Fixed-Source Synthesis (FSS), which holds the source fixed and scales the generation budget. Existing scaling studies typically expand the source as the data grows, conflating SE with FSS and leaving FSS underexplored. We isolate FSS by holding the seed-question pool and teacher model fixed, varying only the per-question response budget under Rejection Sampling (RS). We adapt the rectified scaling law to FSS, deriving it from how repeated sampling covers a fixed source. Empirically, the derived form, fit on low budgets, predicts performance at the held-out highest budget for every evaluated teacher--student pair. At matched total-sample budgets, SE and FSS are comparable at small budgets; at large budgets, adding seed questions outperforms spending the same budget on more responses. Within FSS, however, neither synthesizing additional questions from the existing seeds nor varying the synthesis protocol outperforms plain RS at matched budgets. FSS is thus a bounded scaling axis and a controlled setting for comparing synthesis protocols. We will release our code and data to facilitate further research.
