Search papers, labs, and topics across Lattice.
The paper introduces a novel method for controllable image generation by composing multiple input conditions in discrete generative models. They derive a theoretically-grounded formulation for composing discrete probabilistic generative processes, enabling precise control over the combination and weighting of input conditions. Experiments on CLEVR and FFHQ datasets demonstrate a significant reduction in error rate and improvement in FID score compared to state-of-the-art methods, along with substantial speedups.
Achieve fine-grained control over image generation with a method that's not only 63% more accurate and 2.3x-12x faster, but also lets you precisely dial in the influence of each input condition.
Conditional discrete generative models struggle to faithfully compose multiple input conditions. To address this, we derive a theoretically-grounded formulation for composing discrete probabilistic generative processes, with masked generation (absorbing diffusion) as a special case. Our formulation enables precise specification of novel combinations and numbers of input conditions that lie outside the training data, with concept weighting enabling emphasis or negation of individual conditions. In synergy with the richly compositional learned vocabulary of VQ-VAE and VQ-GAN, our method attains a $63.4\%$ relative reduction in error rate compared to the previous state-of-the-art, averaged across 3 datasets (positional CLEVR, relational CLEVR and FFHQ), simultaneously obtaining an average absolute FID improvement of $-9.58$. Meanwhile, our method offers a $2.3\times$ to $12\times$ real-time speed-up over comparable methods, and is readily applied to an open pre-trained discrete text-to-image model for fine-grained control of text-to-image generation.
