Search papers, labs, and topics across Lattice.
This paper introduces Off-Context GRPO (OC-GRPO), a novel approach that enhances reinforcement learning with verifiable rewards (RLVR) by incorporating privileged information during training. By utilizing off-context rollouts generated from guided prompts, OC-GRPO effectively mitigates the learning cliff encountered in challenging reasoning tasks, allowing models to receive meaningful learning signals even when initial outputs are incorrect. The method demonstrates a significant 3.9% absolute improvement over traditional GRPO on mathematical reasoning benchmarks, showcasing its efficacy with minimal additional computational cost.
Off-Context GRPO achieves a 3.9% absolute improvement in reasoning tasks by leveraging privileged information to guide learning, even when models struggle to produce correct outputs.
Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives \textit{zero} learning signal. Providing privileged guidance during training, such as solution prefixes, can help overcome this learning cliff by steering the model towards {correct solutions with non-zero reward}. {We call these rollouts \textit{off-context}: they are generated from a training prompt that contains privileged guidance, while the target objective is defined by the original prompt without that guidance.} {We introduce} Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training. Empirically, our algorithm achieves a 3.9\% absolute improvement (13.8\% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks with negligible additional cost.
Meta AI