Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Hindsight-Privileged RLVR (HP-RLVR) addresses the challenge of assigning token-level credit in Reinforcement Learning from Verifiable Rewards (RLVR) by leveraging a trajectory's own future as a hindsight context. This single-rollout method generates token-level update weights by comparing a token's score under its causal history against its score under a hindsight context incorporating the realized future and verifier feedback, normalized by Jensen-Shannon divergence. HP-RLVR achieves significantly faster training and more consistent gradient signals compared to group-sampling methods like GRPO across various tasks, with comparable or improved accuracy.
A single-rollout RLVR method achieves faster training and more stable gradients by using a trajectory's own future as a hindsight context, eliminating the need for group sampling and reference solutions.
Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared $100$-step schedule, HP-RLVR trains $1.21\times$, $1.46\times$, and $2.30\times$ faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, $43.2\%$ (4B) and $49.0\%$ (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.