Search papers, labs, and topics across Lattice.
The paper introduces Evolutionary System Prompt Learning (E-SPL), a method that jointly optimizes LLM weights via RL and system prompts via evolutionary algorithms. E-SPL leverages LLM-driven mutation and crossover to evolve a population of system prompts, selecting prompts based on TrueSkill ratings derived from their performance within each RL iteration. Results demonstrate that E-SPL improves both sample efficiency and generalization, achieving a 45.1% success rate in an AIME $\rightarrow$ BeyondAIME generalization setting, outperforming reflective prompt evolution (40.0%).
LLMs learn better when you evolve their system prompts alongside their weights, boosting generalization in reasoning tasks.
Building agentic systems that can autonomously self-improve from experience is a longstanding goal of AI. Large language models (LLMs) today primarily self-improve via two mechanisms: self-reflection for context updates, and reinforcement learning (RL) for weight updates. In this work, we propose Evolutionary System Prompt Learning (E-SPL), a method for jointly improving model contexts and model weights. In each RL iteration, E-SPL selects multiple system prompts and runs rollouts with each in parallel. It applies RL updates to model weights conditioned on each system prompt, and evolutionary updates to the system prompt population via LLM-driven mutation and crossover. Each system prompt has a TrueSkill rating for evolutionary selection, updated from relative performance within each RL iteration batch. E-SPL encourages a natural division between declarative knowledge encoded in prompts and procedural knowledge encoded in weights, resulting in improved performance across reasoning and agentic tasks. For instance, in an easy-to-hard (AIME $\rightarrow$ BeyondAIME) generalization setting, E-SPL improves RL success rate from 38.8% $\rightarrow$ 45.1% while also outperforming reflective prompt evolution (40.0%). Overall, our results show that coupling reinforcement learning with system prompt evolution yields consistent gains in sample efficiency and generalization. Code: https://github.com/LunjunZhang/E-SPL
