Search papers, labs, and topics across Lattice.
This paper introduces Group Entropy-Controlled Policy Optimization (GEPO), a novel approach that enhances the reinforcement learning of large language models by addressing the challenges posed by heterogeneous task mixtures. By employing group entropy for asymmetric advantage shaping, GEPO effectively balances exploration and exploitation across varying entropy regimes, mitigating biases seen in traditional methods. Experimental results demonstrate that GEPO outperforms existing techniques, achieving superior performance across diverse benchmarks while maintaining appropriate exploration levels for specific tasks.
GEPO achieves balanced cross-task improvements in RL for LLMs by dynamically adjusting advantages based on group entropy, outperforming traditional methods.
Entropy control has become an effective tool in reinforcement learning (RL) of large language models (LLMs), helping balance exploration-exploitation trade-off during alignment process. Such RL paradigm is often conducted on mixtures of heterogeneous tasks, which induce distinct entropy regimes under the same policy, making global or token-level entropy regulation insufficient to corresponding heterogeneous needs of exploration. This heterogeneity further makes GRPO-style normalized advantages induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable. To address this issue, we propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy, estimated from existing grouped samples to perform entropy-conditioned asymmetric advantage shaping. GEPO attenuates positive advantages in low-entropy groups to reduce over-exploitation, and negative advantages in high-entropy groups to preserve exploration, with adaptive thresholds derived from historical entropy statistics. Extensive experiments on two base models across thirteen benchmarks spanning mathematics, physics, science, code generation, and instruction following show that GEPO consistently outperforms GRPO and recent entropy-controlled methods, delivering balanced cross-task improvements while preserving task-specific exploration levels throughout training.
