Search papers, labs, and topics across Lattice.
This paper introduces OP-GRPO, an off-policy variant of GRPO designed to improve the sample efficiency of flow-matching model fine-tuning. OP-GRPO employs a replay buffer with adaptive trajectory selection, sequence-level importance sampling to correct for distribution shift, and trajectory truncation to address ill-conditioned importance ratios in late denoising steps. Experiments on image and video generation tasks demonstrate that OP-GRPO achieves comparable or better performance than on-policy Flow-GRPO while using only 34.2% of the training steps.
Flow-matching models can now be fine-tuned with GRPO using only a third of the training steps, thanks to a novel off-policy approach.
Post training via GRPO has demonstrated remarkable effectiveness in improving the generation quality of flow-matching models. However, GRPO suffers from inherently low sample efficiency due to its on-policy training paradigm. To address this limitation, we present OP-GRPO, the first Off-Policy GRPO framework tailored for flow-matching models. First, we actively select high-quality trajectories and adaptively incorporate them into a replay buffer for reuse in subsequent training iterations. Second, to mitigate the distribution shift introduced by off-policy samples, we propose a sequence-level importance sampling correction that preserves the integrity of GRPO's clipping mechanism while ensuring stable policy updates. Third, we theoretically and empirically show that late denoising steps yield ill-conditioned off-policy ratios, and mitigate this by truncating trajectories at late steps. Across image and video generation benchmarks, OP-GRPO achieves comparable or superior performance to Flow-GRPO with only 34.2% of the training steps on average, yielding substantial gains in training efficiency while maintaining generation quality.
