Search papers, labs, and topics across Lattice.
This paper introduces Flow Advantage-Weighted Rectification (FlowAWR), a novel approach to aligning generative flow models in continuous spaces through online reinforcement learning. By recasting policy optimization as supervised regression towards an optimal velocity field, FlowAWR eliminates the need for stochastic SDE samplers and Classifier-Free Guidance, addressing training-inference inconsistencies. Experimental results demonstrate that FlowAWR not only accelerates convergence by 2 to 5 times compared to DiffusionNFT but also achieves superior alignment performance and maintains generation quality under multi-reward constraints.
FlowAWR accelerates convergence in generative flow models by up to 5 times while ensuring superior alignment and quality under complex reward structures.
Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-inference inconsistencies and necessitates Classifier-Free Guidance (CFG). While implicit frameworks such as DiffusionNFT directly optimize forward-process velocity fields, its heuristic fixed-magnitude corrections prevent optimization strength from relative intra-group quality. We propose \textit{Flow Advantage-Weighted Rectification} (\textbf{FlowAWR}), a paradigm that recasts continuous generative policy optimization as supervised regression toward a theoretically optimal velocity field. Starting from the optimal policy of a KL-constrained reward maximization, FlowAWR derives the optimal velocity field that admits a magnitude-aware, advantage-weighted rectification form, yielding SDE-free optimization and CFG-free generation. In comparative evaluations on SD3.5-Medium, FlowAWR achieves improved alignment performance alongside a 2$\times$ to 5$\times$ convergence acceleration over DiffusionNFT (e.g., reaching a 24.12 PickScore in 1.2k steps, versus 23.82 in 2.0k steps for DiffusionNFT and 23.50 in $>$4k steps for FlowGRPO). Under multi-reward constraints, FlowAWR sustains generation quality, satisfying structural rules while maintaining stable out-of-domain performance.
