Search papers, labs, and topics across Lattice.
This paper introduces Dualin, a two-stage method for prompt inversion in text-to-image diffusion models that addresses the limitations of existing techniques by jointly recovering both the semantic prompt and latent noise of target images. The first stage employs CLIP and a large language model to generate a human-interpretable hard prompt, while the second stage utilizes unconditional DDIM inversion to reconstruct the latent noise, ensuring structural consistency. Experimental results show that Dualin achieves high-quality prompt generation and state-of-the-art image fidelity, paving the way for precise and controllable image editing.
Dualin achieves state-of-the-art image fidelity by simultaneously recovering semantic prompts and latent noise, overcoming the limitations of traditional prompt inversion methods.
Prompt inversion, as a typical reverse engineering technique, enables text-to-image (T2I) diffusion models to generate the desired target images without extensive prompt engineering. However, existing prompt inversion methods suffer from significant limitations: (1) gradient-based methods are unstable and uninterpretable, often resulting in generated images with severe artifacts; (2) gradient-free methods yield human-readable prompts but still fail to preserve visual fidelity due to the lack of fine-grained detail alignment. We contend that the limitations stem from treating prompt inversion as a sufficient condition for reverse engineering, ignoring the critical role of the latent noise that encodes structural information. Consequently, we propose Dualin (Dual inversion), a two-stage method that jointly recovers both the semantic prompt and latent noise of the target image. In the first stage, we integrate vision-language model, CLIP and large language model to invert a faithful, human-interpretable hard prompt. In the second stage, unconditional DDIM inversion reconstructs the exact latent noise of the target image, guaranteeing the consistency at the structural information level. Theoretically, we prove that the inverted noise enables flexible image editing without re-optimization. Extensive experiments on diverse datasets demonstrate that Dualin simultaneously generates high-quality inverted prompts and achieves state-of-the-art image fidelity. Additionally, Dualin can establish a robust foundation for the precise and controllable image editing.
