Search papers, labs, and topics across Lattice.
This paper explores training-time data augmentation techniques to address overfitting in autoregressive (AR) language model pretraining under data-constrained conditions. By systematically investigating three categories of augmentation鈥攖oken-level noise, sequence permutations, and target offset prediction鈥攖he authors demonstrate that these methods can significantly delay overfitting and reduce validation loss during extended training on fixed corpora. The findings reveal that combining multiple augmentation strategies yields the lowest minimum validation loss, offering a robust solution for enhancing training efficiency in resource-limited scenarios.
Training-time data augmentation can slash validation loss in language model pretraining, making it feasible to train effectively on limited datasets for hundreds of epochs.
As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora. Standard autoregressive (AR) pretraining overfits severely in this setting, reaching its optimum early and then continuously deteriorating. We investigate training-time data augmentation as a regularizer to mitigate this overfitting and enable productive training for hundreds of epochs on the same data. We introduce three orthogonal categories of augmentation for AR pretraining: token-level noise (masking, random replacement), sequence permutations (right-to-left prediction, Fill-in-the-Middle), and target offset prediction (x_{t+i} for i > 1). Through systematic ablations, we find that individual augmentations delay overfitting and lower validation loss relative to the baseline, with random token replacement achieving the best minimum loss among individual methods. Combining augmentation categories further lowers the minimum validation loss. Our experiments demonstrate that data augmentations mitigate AR pretraining's data inefficiency and offer a promising solution to the data-constrained regime~\footnote{All code and data are available at https://github.com/ michaelchen-lab/ data-augmentations-for-pretraining.
