Search papers, labs, and topics across Lattice.
This paper addresses the exploration collapse issue in reinforcement learning (RL) for large language models (LLMs) caused by the limitations of PPO-Clip, which uses an inconsistent Euclidean metric for policy discrepancy. By introducing Riemannian Isometric Policy Optimization (RIPO), the authors correct this geometric mismatch, ensuring isometric updates on the Riemannian manifold that effectively balance exploration and exploitation. Experimental results show that RIPO outperforms existing RL algorithms for LLMs, achieving up to a 60% improvement over the state-of-the-art GRPO on competitive benchmarks.
RIPO transforms LLM reinforcement learning by correcting geometric flaws, leading to up to 60% better performance on key benchmarks.
Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs'reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip's failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).
Tsinghua AI