Search papers, labs, and topics across Lattice.
This paper investigates catastrophic forgetting in LLM fine-tuning with Evolution Strategies (ES), finding that it's better characterized as performance drift that recovers during training and is not unique to ES. They identify that drift arises from random walk behavior in weakly constrained directions of the weight space during ES training. To mitigate this, they introduce Anchored Weight Decay (AWD), a parameter-space regularization technique that constrains optimization toward the initial model parameters, effectively stabilizing prior-task performance.
Catastrophic forgetting in LLM fine-tuning with Evolution Strategies isn't irreversible or unique to ES, and can be largely avoided with a simple regularization technique.
Evolution Strategies (ES) has recently emerged as a competitive alternative to reinforcement learning (RL) for large language model (LLM) fine-tuning, offering advantages through simplicity, scalability, and inference-only training. However, recent work suggests that ES fine-tuning on new tasks may induce forgetting of prior tasks. First, this paper shows that prior task forgetting (1) is better characterized as performance drift rather than irreversible forgetting, with prior-task performance often recovering during ES training; and (2) is not a specific failure mode of ES, but can also arise for fine-tuning with RL methods. Second, it analyzes when and why such drift arises, highlighting its dependence on ES training dynamics, particularly random walk behavior in weakly constrained directions of the weight space. Third, based on these insights, it introduces Anchored Weight Decay (AWD) as a parameter-space regularization technique that constrains optimization toward the initial model parameters. AWD effectively stabilizes prior-task performance while preserving target-task performance, achieving benefits comparable to large ES population sizes at much lower computational cost. Thus, contrary to previous beliefs, the paper shows that prior-task forgetting under ES is largely avoidable, positioning ES as a promising approach for continual learning in LLMs.
