Search papers, labs, and topics across Lattice.
This paper introduces In-Loop Model Adaptation (IMA), a novel approach for enhancing text-to-image diffusion models by adapting the model at each generation step rather than relying on extensive pre-training with reference images. By employing a DDIM inversion chain and a coupled latent-noise loss, IMA effectively preserves subject identity while aligning with varying visual contexts specified by text prompts. Experimental results show that IMA significantly outperforms existing methods in generating high-fidelity images that accurately reflect both the subject and the textual description.
IMA achieves high-fidelity text-to-image generation by adapting the model in real-time, preserving subject identity without pre-training on reference images.
Text-to-image diffusion models have achieved remarkable success in generating high-quality images from a given text prompt. Subject-driven generation aims to synthesize customized images to mimic the appearance of subjects in given reference images within different visual contexts specified by the text prompts. The central challenge here is that, when the reference image changes, the diffusion model cannot efficiently adapt to different visual contexts while consistently maintaining the subject identity. Existing methods either train the model with a large domain-specific dataset or fine-tune the model using the reference image for hundreds of iterations before actual image generation. In this work, we explore a new approach, called \textit{In-Loop Model Adaptation} (IMA), which adapts the core diffusion model at each generation step during the actual process of image generation, without being trained on the reference image before the generation process. To this end, we establish a DDIM inversion chain that maps the reference image to a sequence of latent, as well as a text-to-image generation chain which generates the image from the text prompt only. We then introduce a masked latent consistency loss and a noise regularization loss to characterize the latent-noise difference between the diffusion model and these two chains at each generation step. This coupled latent-noise loss is used to guide the in-loop model adaptation to preserve the subject identity specified by the reference image while maintaining accurate alignment with the text prompt, resulting in high-fidelity text-to-image generation. Our extensive experiments demonstrate that our proposed IMA method significantly improves the performance of subject-driven text-to-image generation.
