Search papers, labs, and topics across Lattice.
This paper addresses the challenge of multi-task reinforcement learning (RL) in agentic large language models (LLMs) by introducing Entropy Pacing Policy Optimization (EPPO), which stabilizes optimization across tasks with varying exploration-exploitation dynamics. The authors identify that easier tasks can prematurely converge to low-entropy policies, negatively impacting the learning of harder tasks, leading to detrimental inter-task interactions. Through a task-wise dynamic clipping mechanism, EPPO adapts the entropy bounds for each task, resulting in superior performance on multi-task benchmarks compared to existing methods.
Easier tasks can sabotage the learning of harder tasks in multi-task RL, but a new entropy-aware optimization strategy can turn this challenge into an advantage.
Recent breakthroughs of Reinforcement Learning (RL) have highlighted its potential for complex agentic Large Language Model (LLM) tasks. However, existing efforts largely focus on single-task settings, whereas real-world deployment necessitates a generalist agent capable of solving multiple tasks simultaneously. In this work, we identify a critical yet underexplored phenomenon in multi-task agentic RL: different tasks can exhibit exploration-exploitation pace mismatch. Specifically, easier tasks may converge early to low-entropy policies that hinder learning on harder tasks, while harder tasks can, in turn, push easier tasks back toward high-entropy exploration. This back-and-forth interaction creates inter-task entropy crossovers and frequent entropy spikes. Inspired by this observation, we introduce Entropy Pacing Policy Optimization (EPPO) for multi-task agentic LLMs, which coordinates entropy across tasks to stabilize multi-task optimization. At the core of EPPO is a task-wise dynamic clipping mechanism that replaces the fixed clipping threshold in Group Relative Policy Optimization (GRPO) with a task entropy-aware adaptive bound, tightening updates for over-confident tasks while relaxing them for under-explored ones. Experiments on the multi-task agentic benchmarks demonstrate that the proposed EPPO yields results superior to its counterparts.
