Search papers, labs, and topics across Lattice.
This paper explores the concept of verifier-induced support reshaping in on-policy reinforcement learning with verifiable rewards (RLVR), revealing that while RLVR can enhance the current objective, it can also lead to a scarcity of successful behaviors for future objectives. The authors demonstrate that this phenomenon results in a trade-off where average success rates improve, but the diversity of successful responses diminishes, particularly in mathematical reasoning and instruction-following tasks. Through empirical evaluation, they show that while RLVR can boost performance metrics, it does not guarantee the retention of trainability or capability across multiple tasks due to the concentration of changes in initial response tokens.
Enhancing on-policy reinforcement learning can paradoxically reduce the diversity of successful behaviors, leading to a trade-off that challenges future trainability.
We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce. We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget. Across two model families, we study this effect through repeated verifier-scored sampling and bidirectional training on mathematical reasoning and constrained instruction following, including sequential training with the opposite verifier. Math-RLVR raises average instruction-following success but reduces the number of prompts with any successful response under repeated sampling. On IFEval with Qwen3-8B-Base, pass@1 rises by 6.5 percentage points while best@32 falls by 9.8 percentage points, and the same divergence appears across both models and IF benchmarks. Conversely, IF-RLVR shifts math responses from step-by-step openings toward direct answers, lowers best@k across sampling budgets, and reduces reward variation for later Math-RLVR. Token-distribution analyses and controlled opening interventions show that these changes concentrate in the first few response tokens. RLVR mainly reranks openings already available in the base policy, and the selected opening causally affects math searchability. The tested reference-policy constraints, routing priors, and on-policy distillation preserve cross-task support only partially; MathIF and ReasonIF show that marginal gains translate only partly into responses that are both correct and constraint-following. Therefore, endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization. Code is available at https://github.com/sylvain-wei/verifier-induced-support-reshaping
