Search papers, labs, and topics across Lattice.
This paper analyzes the training dynamics of controllable text-to-image diffusion models and finds that directly supervising the clean target image ($x_0$-supervision) leads to faster convergence. They show that $x_0$-supervision, or an equivalent re-weighting of the diffusion loss, accelerates convergence by up to 2x. Experiments across various control settings demonstrate improved visual quality and conditioning accuracy using the proposed approach.
Controllable diffusion models can be trained twice as fast and achieve better image quality simply by directly supervising the clean target image during training.
Text-to-Image (T2I) diffusion/flow models have recently achieved remarkable progress in visual fidelity and text alignment. However, they remain limited when users need to precisely control image layouts, something that natural language alone cannot reliably express. Controllable generation methods augment the initial T2I model with additional conditions that more easily describe the scene. Prior works straightforwardly train the augmented network with the same loss as the initial network. Although natural at first glance, this can lead to very long training times in some cases before convergence. In this work, we revisit the training objective of controllable diffusion models through a detailed analysis of their denoising dynamics. We show that direct supervision on the clean target image, dubbed $x_0$-supervision, or an equivalent re-weighting of the diffusion loss, yields faster convergence. Experiments on multiple control settings demonstrate that our formulation accelerates convergence by up to 2$\times$ according to our novel metric (mean Area Under the Convergence Curve - mAUCC), while also improving both visual quality and conditioning accuracy. Our code is available at https://github.com/CEA-LIST/x0-supervision
