Search papers, labs, and topics across Lattice.
Standard group relative policy optimization (GRPO) relies on static importance-sampling clipping, which disproportionately suppresses strong gradient signals from rare successful rollouts on challenging prompts. To fix this, Group Adaptive Clipping Policy Optimization (GAPO) dynamically scales the clipping boundary according to rollout advantage, guided by a reverse-KL trust-region framework that allocates greater update headroom to high-signal trajectories. Across Qwen and Llama backbones on competitive math and code benchmarks, GAPO consistently outperforms standard fixed-clipping and advantage-shaping baselines on both Pass@1 and Pass@k.
Standard GRPO's uniform clipping boundary actively chokes exploration by penalizing rare breakthroughs on hard problems just as harshly as trivial rollouts on easy ones.
Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.
