Search papers, labs, and topics across Lattice.
AdvantageFlow is introduced as a forward-process reinforcement learning algorithm for rectified flow models, optimizing an advantage-weighted forward-process prediction loss. To address instability caused by negative advantages and non-convex loss, the method employs rollout policy regularization, effectively reducing variance. Experiments on image generation with Stable Diffusion 3.5 Medium demonstrate that AdvantageFlow outperforms Flow-GRPO and a negative-aware fine-tuning baseline.
Stabilizing advantage-weighted forward-process RL in flow models unlocks superior performance in image generation compared to reverse-process methods.
We introduce AdvantageFlow, a forward-process reinforcement learning algorithm for rectified flow models. Unlike Flow-GRPO, which optimizes the reverse process, we optimize an advantage-weighted forward-process prediction loss. This optimization problem is unstable when advantages are negative and the loss becomes non-convex. We stabilize it by rollout policy regularization, which reduces variance and arises from fitting a local reward-improving target distribution. We evaluate AdvantageFlow on image generation tasks with Stable Diffusion 3.5 Medium. It outperforms both Flow-GRPO and a state-of-the-art forward-process RL baseline based on negative-aware fine-tuning.
