Search papers, labs, and topics across Lattice.
This paper investigates the optimization dynamics of Evolution Strategies (ES) as a post-training paradigm for large language models (LLMs), revealing its superior performance compared to Group Relative Policy Optimization (GRPO). The authors demonstrate that ES achieves broader reasoning coverage, leading to improved Pass@K metrics while avoiding the entropy collapse seen in GRPO. Additionally, they introduce a hybrid GRPO-ES training strategy that leverages the strengths of both methods, highlighting the functional sparsity of updates in ES that contributes to task performance without catastrophic forgetting.
ES not only outperforms GRPO in reasoning tasks but also reveals that substantial parameter updates don't equate to widespread functional changes in LLMs.
Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.
