Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper extends functional scaling laws to incorporate data quality, deriving an optimal data scheduling strategy for LLM training in closed form. The analysis reveals two distinct regimes (noise-limited and signal-limited) and suggests high-quality data can act as either a signal amplifier or noise suppressor. Based on these insights, the authors propose a "Drop-Stable-Rampup" schedule, which outperforms standard schedules like Warmup-Stable-Decay and Cosine-decay, particularly on mathematical reasoning tasks, when midtraining a 15B MoE model.
Forget curriculum learning – the secret to squeezing more out of high-quality data is to *drop* your batch size when you start using it, not just place it late in training.
High-quality data is scarce in large language model (LLM) training, yet how to schedule its use jointly with training dynamics lacks theoretical guidance. We extend functional scaling laws by incorporating a data-quality dimension, and solve the joint data-quality and batch-size scheduling problem in asymptotic closed form. The solution reveals two regimes and a dual role of high-quality data. In the noise-limited regime, high-quality data should be used as a signal amplifier: lowering the batch size converts cleaner data into more signal without amplifying noise. In the signal-limited regime, it should be used as a noise suppressor: late placement reduces terminal noise without sacrificing signal accumulation. Existing curriculum-style pipelines primarily exploit the second role by placing cleaner data late, but miss the first role because conventional decay schedules reduce update intensity exactly when high-quality data becomes available. Guided by this, we propose Drop-Stable-Rampup for LLM midtraining: upon the quality transition, drop the batch size, hold it stable to accumulate signal, then ramp up to suppress terminal noise. On a 15B Mixture-of-Experts model midtrained on 108B tokens, Drop-Stable-Rampup improves average accuracy over Warmup-Stable-Decay (WSD) by +1.70 and over Cosine-decay by +2.98, with particularly large gains on mathematical reasoning benchmarks such as GSM8K (+4.23) and MATH (+2.80).