Search papers, labs, and topics across Lattice.
This paper introduces "Memorization-guided Data Reuse," a novel training paradigm for large language models that leverages loss retention dynamics and downstream evaluation scores to optimize data reuse during multi-epoch training. By identifying a "Memorization Window," the approach allows for adaptive decisions on the number of training epochs and data replay scheduling, significantly enhancing model performance without the pitfalls of overfitting. Preliminary experiments show that models can benefit from data repetition well beyond the traditional four-epoch limit, suggesting a more efficient training strategy with high-quality data.
Performance can improve significantly with data reuse beyond the traditional limits, challenging the status quo of LLM training practices.
The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency. Meanwhile, excessive repetition introduces the risk of overfitting and diminishing returns. Determining when and how to reuse data effectively thus emerges as a natural but under-explored question. Through a novel observation of model's"Memorization Window"signals derived from loss retention dynamics and downstream evaluation scores, we propose"Memorization-guided Data Reuse", a training paradigm that adaptively determines when and how data should be reused, enabling principled decisions on the number of training epochs and the scheduling of data replays. Our preliminary experiments reveal a consistent memorization-driven regime: performance continues to improve with repetition far beyond current practice (e.g., the commonly cited four-epoch limit). While a full scheduler remains future work, these insights provide a foundation for memorization-aware training schedules, helping to determine reuse budgets and move toward training LLMs smarter rather than longer with limited high-quality data.
