Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To eliminate dead-gradient steps caused by all-fail rollout groups in RLVR methods like GRPO, this work introduces GRAFT, an off-policy framework that swaps failed rollout groups with trajectory groups generated by peer models. The method enables mutually beneficial co-training between heterogeneous models without requiring a superior teacher, counteracting cross-policy distribution shift using sequence-level compatibility weighting and token-level importance clipping. Across five mathematical reasoning benchmarks, GRAFT outperforms GRPO by up to 4.5 points under identical rollout budgets and preserves an average 1.8-point gain even when peer trajectories are consumed asynchronously from offline storage.
Heterogeneous LLMs can fix each other's dead-end zero-reward rollouts in RLVR through off-policy trajectory swapping, boosting math reasoning by up to 4.5 points without extra compute or a stronger teacher model.
Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model's rollouts may already have been discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary prompts, creating opportunities for mutual learning without a designated stronger teacher. To exploit this complementarity, we propose GRAFT (Gated Replacement of Answer-Failed groups with peer Trajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups. GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping. Across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT consistently improves both models over GRPO with the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve most of the gains, improving over GRPO by 1.8 points on average without simultaneous co-training.