Search papers, labs, and topics across Lattice.
This paper investigates the impact of data organization strategies on LLM training efficiency, focusing on single-epoch or few-epoch training scenarios. It formalizes four guidelines鈥擝oundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity鈥攆or optimizing data ordering. Based on these guidelines, the authors introduce two novel data ordering methods, STR and SAW, and demonstrate their effectiveness in improving LLM training stability and performance across various model scales and data sizes in both pre-training and SFT.
Forget data selection鈥攔eordering your existing dataset using these four simple guidelines can significantly boost LLM training performance and stability.
Large Language Models (LLMs) have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation. While data selection has been widely studied, the strategic data organization for enhanced training remains an underexplored area, particularly since current LLMs are often trained for only one or a few epochs. This paper systematically explores the influence of data organization on LLM training by reusing pre-computed sample-level scores originally generated for data efficiency, thereby incurring minimal additional computational overhead. We identify and formalize four key guidelines for optimizing data organization: Boundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity. Guided by them, we introduce two novel data ordering methods termed STR and SAW. Extensive experiments across different model scales and data sizes, encompassing both pre-training and SFT stages, validate the effectiveness of our summarized guidelines. They also demonstrate the robustness of our proposed data ordering methods in enhancing the stability and performance of LLM training. Github Link: https://github.com/microsoft/data-efficacy/
Microsoft Research