Search papers, labs, and topics across Lattice.
This paper introduces Modulated Hazard-aware Policy Optimization (MHPO), a novel reinforcement learning framework designed to improve training stability by addressing limitations in importance ratio control. MHPO uses a Log-Fidelity Modulator (LFM) to bound and differentiate importance ratios, preventing high-variance outliers, and a Decoupled Hazard Penalty (DHP) to regulate positive and negative policy shifts independently. Experiments across text-based and vision-language reasoning tasks show MHPO outperforms existing methods with enhanced training stability.
Stabilizing RL training is now possible by modulating importance ratios with hazard-aware penalties, preventing both mode collapse and policy erosion.
Regulating the importance ratio is critical for the training stability of Group Relative Policy Optimization (GRPO) based frameworks. However, prevailing ratio control methods, such as hard clipping, suffer from non-differentiable boundaries and vanishing gradient regions, failing to maintain gradient fidelity. Furthermore, these methods lack a hazard-aware mechanism to adaptively suppress extreme deviations, leaving the optimization process vulnerable to abrupt policy shifts. To address these challenges, we propose Modulated Hazard-aware Policy Optimization (MHPO), a novel framework designed for robust and stable reinforcement learning. The proposed MHPO introduces a Log-Fidelity Modulator (LFM) to map unbounded importance ratios into a bounded, differentiable domain. This mechanism effectively prevents high-variance outlier tokens from destabilizing the loss landscape while ensuring global gradient stability. Complementarily, a Decoupled Hazard Penalty (DHP) integrates cumulative hazard functions from survival analysis to independently regulate positive and negative policy shifts. By shaping the optimization landscape with hazard-aware penalties, the proposed MHPO achieves fine-grained regulation of asymmetric policy shifts simultaneously mitigating mode collapse from over-expansion and preventing policy erosion from catastrophic contraction within a stabilized trust region. Extensive evaluations on diverse reasoning benchmarks across both text-based and vision-language tasks demonstrate that MHPO consistently outperforms existing methods, achieving superior performance while significantly enhancing training stability.
