Search papers, labs, and topics across Lattice.
This paper introduces MDP-GRPO, a novel approach to stabilize group-relative policy optimization (GRPO) in reinforcement learning for multi-constraint instruction following, addressing the instability caused by homogeneous reward distributions. The authors identify three critical pathologies in z-score normalization that hinder performance and propose solutions such as multi-temperature sampling and dual-anchor advantages to enhance reward dispersion and gradient recovery. Evaluated on multiple benchmarks, MDP-GRPO demonstrates a significant improvement in constraint satisfaction and stable convergence, outperforming standard GRPO methods.
MDP-GRPO improves constraint satisfaction in reinforcement learning by up to 5.0% while ensuring stable convergence even with small group sizes.
Reinforcement learning with verifiable rewards is ideal for multi-constraint instruction following, yet standard group-relative policy optimization (GRPO) becomes unstable under discrete, low-dispersion rewards, where within-group reward distributions are frequently homogeneous. We identify and formalize three pathologies of z-score group normalization in this regime: low-variance amplification, mean-centering blindness, and zero-variance collapse. To address them, we propose MDP-GRPO, which stabilizes learning through (1) multi-temperature sampling to increase reward dispersion, (2) dual-anchor advantages to restore gradients in homogeneous groups and stop mean-centering blindness, (3) prospect-theoretic shaping to bound updates and penalize violations based on Kahneman and Tversky's theory, and (4) asymmetric KL regularization. Evaluated on FollowBench, IFEval, and a curated multi-constraint dataset, MDP-GRPO outperforms standard GRPO, improving strict constraint satisfaction by up to 5.0% on Llama-3.2-3B. Our method also enables stable convergence with small group sizes while preserving general capabilities on MMLU and ARC.
