Search papers, labs, and topics across Lattice.
The paper introduces Variance Minimisation Policy Optimisation (VMPO) for diffusion alignment, framing the process as Sequential Monte Carlo and minimizing the variance of log importance weights instead of using a KL divergence objective. This approach is motivated by the SMC interpretation of diffusion alignment where the denoising model acts as a proposal and reward guidance induces importance weights. The authors demonstrate that minimizing the variance objective leads to the reward-tilted target distribution and recovers existing KL-based alignment methods under specific conditions, while also suggesting novel alignment strategies.
Minimizing variance of importance weights in diffusion models elegantly unifies existing alignment methods and opens doors to novel, non-KL optimization strategies.
Diffusion alignment adapts pretrained diffusion models to sample from reward-tilted distributions along the denoising trajectory. This process naturally admits a Sequential Monte Carlo (SMC) interpretation, where the denoising model acts as a proposal and reward guidance induces importance weights. Motivated by this view, we introduce Variance Minimisation Policy Optimisation (VMPO), which formulates diffusion alignment as minimising the variance of log importance weights rather than directly optimising a Kullback-Leibler (KL) based objective. We prove that the variance objective is minimised by the reward-tilted target distribution and that, under on-policy sampling, its gradient coincides with that of standard KL-based alignment. This perspective offers a common lens for understanding diffusion alignment. Under different choices of potential functions and variance minimisation strategies, VMPO recovers various existing methods, while also suggesting new design directions beyond KL.
