Search papers, labs, and topics across Lattice.
This paper addresses the challenge of concept association bias in text-to-image diffusion models, particularly for one-and-only (OAO) objects that resist modification through standard prompting. By employing an information-theoretic analysis, the authors reveal that final text embeddings lose crucial concept-level information, which they recover by injecting intermediate text representations into the denoising process. Their proposed Intermediate Text Representation-guided diffusion method significantly improves alignment with unusual prompts, achieving up to a 19.1 percentage-point increase in VQAScore without additional training or external models.
Injecting intermediate text representations into the denoising process can dramatically enhance the alignment of text-to-image models with challenging prompts for unique objects.
Text-to-image (T2I) diffusion models often fail to faithfully render explicit textual descriptions, instead defaulting to strongly learned visual priors due to a phenomenon referred to as concept association bias. We show that such bias is particularly strong for one-and-only (OAO) objects, entities that exist in a single canonical form, such as celestial bodies, landmarks, and artworks. The deeply ingrained visual identity for these concepts often resists modification through prompting alone. Addressing this challenge, we first identify through an information-theoretic analysis that the final text embedding discards concept-level information present in the intermediate-layer text representations, reducing the mutual information available to the subsequent denoising process. We then propose Intermediate Text Representation (IR)-guided diffusion, which injects intermediate hidden states of the text encoder into the conditioning signal during early denoising steps, recovering suppressed concepts without any additional training, optimization, or external models. To systematically evaluate the challenging task of aligning generative outputs with unusual prompts for OAO objects, we introduce OAO-AttackBench, a benchmark comprising counterfactual prompts that directly conflict with the core visual identity of OAO objects. Experiments on four benchmarks, including OAO-AttackBench, show that our method achieves up to a 19.1 percentage-point improvement in VQAScore while preserving generation fidelity and human preference. Project page: https://soyoun-won.github.io/one-and-only-ir-guidance/.
