Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to reinforcement learning with verifiable rewards (RLVR) by incorporating temporal scheduling of credit allocation during policy optimization. By emphasizing targeted tokens with specific behaviors and gradually shifting focus towards general optimization, the authors achieve more stable and efficient learning dynamics compared to traditional methods. Experimental results across various reasoning benchmarks indicate that this temporal scheduling significantly enhances policy evolution and performance, highlighting its potential as a critical optimization dimension in RLVR.
Scheduling when learning signals are applied can dramatically enhance the stability and efficiency of policy evolution in RLVR.
Reinforcement learning with verifiable rewards (RLVR) has become a core technique for post-training of Large Language Models (LLMs). While policy optimization is driven by all sampled tokens under a globally broadcast scalar reward, the heterogeneous policy behaviors exhibited along trajectories are largely overlooked without differentiation. Existing works address this by credit allocation, including token-level advantage reweighting, and selective token optimization, however, the allocation criterion are principally stagnant throughout training, limiting resilient policy evolution. In this work, we argue that when learning signals are scheduled can be as important as where they are allocated across tokens, and introduce the temporal dimension that scheduling the credit allocation criteria over the course of RLVR optimization. We find that prioritizing targeted tokens emphasized with specific policy behaviors, and gradually attenuating toward general optimization leads to more stable and efficient learning dynamics. Furthermore, we show that simple trajectory percentiles provide a natural perspective for distinguishing policy behaviors, and works effectively with temporal scheduling. Our analysis reveals that standard optimization substantially sacrifices policy entropy when simultaneously accommodating heterogeneous behaviors, whereas temporal scheduling yields healthier policy evolution dynamics. Experiments across mathematical and general reasoning benchmarks demonstrate consistent improvements, suggesting that temporal scheduling constitutes a promising optimization dimension.
