Search papers, labs, and topics across Lattice.
This paper introduces SC-GRPO, a novel approach to reinforcement learning with verifiable rewards that addresses the limitations of existing credit assignment methods by leveraging self-conditioned learning. By conditioning on its own verified trajectories, SC-GRPO effectively assigns per-token credit using KL divergence as a multiplicative weight on GRPO gradients, enhancing the model's ability to focus on pivotal reasoning steps. The method demonstrates significant improvements across five benchmarks, outperforming traditional methods like GRPO and DAPO by 8.1% and 5.9%, respectively, while also achieving superior out-of-distribution performance.
Self-conditioning on verified trajectories boosts reinforcement learning performance by over 8%, revealing the power of internal feedback in credit assignment.
Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps. Existing token-level credit assignment methods require resources beyond the model's own rollouts. GRPO variants rely on process reward models or ground-truth answers. Knowledge distillation assigns credit through per-token divergence but requires external teachers (On-Policy Distillation) or privileged information (On-Policy Self Distillation). However, these dependencies limit applicability in the pure RLVR setting. We observe that conditioning the model on its own verified trajectories induces a measurable per-token KL divergence between the original and conditioned distributions, and prove that distilling from a self-teacher constructed by verified trajectories leads to infeasible weighted-average solutions when multiple verified trajectories exist. We propose SC-GRPO (Self-Conditioned GRPO), which uses KL divergence mentioned before as a multiplicative weight on GRPO gradients. Across five benchmarks spanning math, code, and agentic tasks, SC-GRPO consistently outperforms 8.1% over GRPO and 5.9% over DAPO with stronger OOD performance. Moreover, SC-GRPO achieves higher performance than OPD.
