Search papers, labs, and topics across Lattice.
This paper introduces Flow-DPPO, a novel approach to Proximal Policy Optimization tailored for flow matching models, addressing the limitations of traditional ratio clipping methods. By utilizing a divergence proximal constraint and leveraging the Gaussian nature of flow model policies for precise KL divergence computation, Flow-DPPO enhances training stability and efficiency. Experimental results demonstrate that Flow-DPPO not only achieves higher rewards but also mitigates issues like catastrophic forgetting and supports balanced multi-objective optimization during training.
Flow-DPPO outperforms traditional PPO methods by achieving higher rewards and greater training stability through a novel divergence proximal constraint.
Recent work has demonstrated that online reinforcement learning (RL) can substantially improve the quality and alignment of flow matching models for image and video generation. Methods such as Flow-GRPO and CPS cast the denoising process as a Markov Decision Process and apply PPO-style ratio clipping to enforce a trust region. However, we argue that ratio clipping is structurally ill-suited for flow models: the probability ratio between new and old policies is a noisy, single-sample estimate of the true policy divergence, leading to over-constraining in some regions of the trajectory and under-constraining in others. We propose Flow-DPPO (Flow Divergence Proximal Policy Optimization), which replaces ratio clipping with a divergence proximal constraint. A key observation is that the per-step policy in flow models is Gaussian, enabling exact and cheap computation of the KL divergence between old and new policies. Flow-DPPO employs an asymmetric divergence mask that blocks gradient updates only when they simultaneously move away from the trusted region and violate the divergence threshold. Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency, alleviates catastrophic forgetting, promotes balanced multi-objective optimization, and enables stable multi-epoch training where ratio clipping degrades. Code and models are available at https://github.com/Tencent-Hunyuan/UniRL/tree/main/FlowDPPO.
NUS