Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the trade-off between alignment and diversity in diffusion policy optimization (DDPO) by theoretically demonstrating that only updating latter timesteps can harm diversity. The authors introduce iADD, an incremental Feynman-Kac training method that achieves superior alignment-diversity performance. Extensive experiments across three tasks validate iADD's significant gains over existing DDPO approaches.
Diffusion models can lose diversity when only later timesteps are updated during reward optimization, a finding that challenges prior assumptions and motivates a new training paradigm.
Reinforcement learning based post training of diffusion models, such as Denoising Diffusion Policy Optimization (DDPO), optimizes a reverse diffusion process under a reward function. However, current approaches to reward optimizations do so at the cost of diversity and quality. In this paper, we provide better tradeoffs through careful theoretical considerations and method design. We analyze the theoretical framework and mathematically demonstrate that \emph{only-latter timestep} updates of diffusion model may be harmful for diversity contrary to the conclusions presented in a previous work. Additionally, we propose an incremental Feynman-Kac training based on strong theoretical foundations in order to achieve the best-yet alignment-diversity tradeoffs. We perform extensive experiments and compare our method against related diffusion policy optimization approaches in three different tasks and also provide strong ablations for each component, thus validating strong performance gains in both alignment and diversity.