Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
Off-policy RLVR suffers from sign-dependent gradient starvation due to clipping, where under-generated positive responses are suppressed while over-generated negative responses dominate. ReSPO addresses this by replacing clipping with a smooth, two-branch kernel derived from an $\alpha$-divergence objective and exponential variance control, preserving positive gradients and suppressing negative ones. Experiments on dense and MoE Qwen3 models show ReSPO accelerates early optimization, improves final scores, and enhances held-out benchmark performance under rollout reuse.
A novel kernel function circumvents gradient starvation in off-policy RLVR, enabling learning from previously discarded positive trajectories.
Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy. We identify a sign-dependent gradient starvation problem in clipped policy optimization: clipping suppresses under-generated positive responses at the low-importance-weight tail while permitting severely over-generated negative responses to dominate the high-weight tail. To address this, we propose ReSPO (Reshaped Sequence Policy Optimization), which replaces clipping with a smooth, two-branch sequence-level kernel derived from an $\alpha$-divergence variational objective and an exponential variance-control tilt. The positive branch preserves a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses. We demonstrate that ReSPO effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail. On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under a rollout reuse, validating our approach on importance-weight tail control in off-policy learning.