Search papers, labs, and topics across Lattice.
This paper investigates catastrophic overtraining in LLMs during supervised fine-tuning (SFT) by examining the role of the learning rate. They show that different learning rates during SFT lead to qualitatively different models despite converging to similar SFT loss values. The key finding is that learning rate decay increases model sharpness, which then exacerbates catastrophic forgetting during SFT, ultimately causing overtraining.
Learning rate decay, a common optimization technique, might be the culprit behind catastrophic forgetting in LLMs during fine-tuning.
Supervised fine-tuning (SFT) is a common first stage of LLM post-training, teaching the model to follow instructions and shaping its behavior as a helpful assistant. At the same time, SFT may harm the fundamental capabilities of an LLM, particularly after long pretraining: a phenomenon known as catastrophic overtraining (Springer et al., 2025). To understand overtraining, we first investigate catastrophic forgetting in finetuning through the lens of implicit regularization of the learning rate. For models trained to the same SFT loss, we identify how the learning rate mediates optimization: finetuning with large and small steps converges to qualitatively different models. Next, we link forgetting to overtraining: learning rate decay increases the sharpness of the pretrained model, which in turn exacerbates catastrophic forgetting during SFT, leading to overtraining. Our findings paint a picture of the overtraining mechanism in LLMs and broadly contribute to the understanding of the interplay between optimization dynamics during pretraining and finetuning.
