Search papers, labs, and topics across Lattice.
This paper introduces MixDiffusion, a novel framework that enables multi-condition text-to-image generation by integrating multiple pre-trained uni-condition diffusion models without requiring additional training. By deriving the predicted noise distribution from various control conditions鈥攕uch as bounding boxes, keypoints, and sketches鈥擬ixDiffusion enhances the flexibility of image synthesis beyond the limitations of existing methods. The key result demonstrates that this approach can seamlessly incorporate diverse control modalities, making it both easy to deploy and extensible for future applications.
MixDiffusion allows for the integration of multiple control conditions in text-to-image generation, breaking the constraints of traditional single-condition models.
Recent advances in text-to-image (T2I) generation have enabled controllable image synthesis by incorporating conditions beyond text. However, most existing diffusion-based methods are limited to a single type of control condition (e.g., bounding boxes or keypoints), which restricts their flexibility. To address this limitation, we propose MixDiffusion, a training-free diffusion framework for multi-condition T2I generation. MixDiffusion theoretically supports an arbitrary number of control conditions, including bounding boxes, keypoints, sketches, depth maps, reference images, and text, by collaboratively integrating multiple pre-trained uni-condition diffusion models. The key insight of the proposed approach is to derive the predicted noise distribution in each denoising step of the diffusion-based multi-condition image generation model from the predicted noise distributions of multiple diffusion-based uni-condition models with a derived integration formula, which is supported by rigorous theory proof. Owing to its training-free nature, MixDiffusion is easy to deploy and readily extensible to new control modalities.
