Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for optimizing visual generative models using distribution-wise rewards instead of conventional sample-wise rewards, addressing issues of reward hacking and visual anomalies. By evaluating the entire data distribution rather than individual samples, the approach effectively mitigates mode collapse and enhances image diversity. Extensive experiments demonstrate significant improvements in FID-50K scores across various models, indicating better alignment with real-world data distributions and improved perceptual quality.
Distribution-wise rewards can drastically enhance image diversity and quality in generative models, reducing mode collapse and reward hacking issues.
Conventional reinforcement learning strategies for visual generation typically employ sample-wise reward functions, yet this practice frequently results in reward hacking that degrades image diversity and introduces visual anomalies. To address these limitations, we present a novel framework that finetunes generative models using distribution-wise rewards, ensuring better alignment with real-world data distributions. Unlike rewards that evaluate samples individually, distribution-wise reward accounts for the data distribution of the samples, mitigating the mode collapse problem that occurs when all samples optimize towards the same direction independently. To overcome the prohibitive computational cost of estimating these rewards, we introduce a subset-replace strategy that efficiently provides reward signals by updating only a small subset of a generated reference set. Additionally, we apply RL to optimize post-hoc model merging coefficients, potentially mitigating the train-inference inconsistency caused by introducing stochastic differential equation (SDE) in regular RL practices. Extensive experiments show our approach significantly improves FID-50K across various base models, from 8.30 to 5.77 for SiT and from 3.74 to 3.52 for EDM2. Qualitative evaluation also confirms that our method enhances perceptual quality while preserving sample diversity.
