Search papers, labs, and topics across Lattice.
The paper introduces Stepwise-Flow-GRPO, a reinforcement learning method for training flow-matching generative models that addresses the limitations of uniform credit assignment in Flow-GRPO. It assigns credit based on the reward improvement at each step of the diffusion process, leveraging Tweedie's formula for intermediate reward estimation and gain-based advantages. Experiments demonstrate improved sample efficiency and faster convergence compared to Flow-GRPO.
Correcting errors early in the diffusion process matters more than fixing them later: Stepwise-Flow-GRPO leverages this insight to dramatically improve RL-based flow model training.
Flow-GRPO successfully applies reinforcement learning to flow models, but uses uniform credit assignment across all steps. This ignores the temporal structure of diffusion generation: early steps determine composition and content (low-frequency structure), while late steps resolve details and textures (high-frequency details). Moreover, assigning uniform credit based solely on the final image can inadvertently reward suboptimal intermediate steps, especially when errors are corrected later in the diffusion trajectory. We propose Stepwise-Flow-GRPO, which assigns credit based on each step's reward improvement. By leveraging Tweedie's formula to obtain intermediate reward estimates and introducing gain-based advantages, our method achieves superior sample efficiency and faster convergence. We also introduce a DDIM-inspired SDE that improves reward quality while preserving stochasticity for policy gradients.
