Search papers, labs, and topics across Lattice.
This paper introduces DRIFT, an innovative online self-evolution policy optimization framework designed to enhance the self-improvement capabilities of large language models in complex reasoning tasks. By employing Difficulty Routing to track learning progress at the problem level and Rhythm Gating to refine token-level policy updates, DRIFT effectively balances exploration and exploitation during training. The framework achieves state-of-the-art performance across five benchmarks, surpassing previous methods by significant margins, particularly excelling in the ToolUse benchmark with a 13.5% improvement over the best prior approach.
DRIFT not only sets a new state-of-the-art in self-improvement for language models but also redefines how we can dynamically adapt learning strategies based on problem difficulty.
Enabling large language models to achieve stable self-improvement without external expert supervision remains a central challenge in complex reasoning tasks. Existing self-distillation and reinforcement learning methods lack explicit mechanisms for tracking problem-level learning progress and adapting optimization strategies accordingly. Consequently, training may over-optimize easy problems, receive weak supervision from hard problems, and fail to sufficiently explore borderline cases. To resolve these issues, we propose DRIFT, an online self-evolution policy optimization framework for large language models. DRIFT regulates the model's self-improvement process through the joint use of Difficulty Routing and Rhythm Gating. The former identifies the model's learning state at the problem level and dynamically allocates self-distillation and reinforcement learning signals, while the latter refines policy updates at the token level, concentrating exploration on critical reasoning positions. By further incorporating a success buffer and a two-stage curriculum learning strategy, DRIFT preserves high-quality historical experience while progressively guiding the model from reliable behavior acquisition toward stable policy evolution. Evaluated across five benchmarks and three model scales, DRIFT surpasses the peak performance of both GRPO and SDPO across all evaluated metrics. On the average score over the five benchmarks, DRIFT achieves 79.5$\%$, outperforming GRPO by 9.5$\%$ and SDPO by 7.5$\%$, establishing a new state-of-the-art result. Notably, on ToolUse, DRIFT reaches an accuracy of 79.2$\%$, improving over GRPO by 13.5$\%$ and SDPO by 10.7$\%$, setting a new state-of-the-art and substantially outperforming all concurrent methods.
Tsinghua AI