Search papers, labs, and topics across Lattice.
The paper introduces DisCo, a framework for subject-driven text-to-image generation that addresses the similarity-controllability paradox by disentangling and then recoupling visual and textual information. DisCo extracts subject identity solely from a reference image using the entity word and simplifies the text prompt to focus only on modification commands, thus removing descriptive ambiguity. A reinforcement learning approach with a dedicated reward signal then seamlessly integrates the visually-defined subject with the textually-generated context, leading to improved subject fidelity and textual control.
Stop sacrificing subject fidelity for editability: DisCo lets you have both in text-to-image generation by disentangling and recoupling visual and textual information.
Subject-Driven Text-to-Image (T2I) Generation aims to preserve a subject's identity while editing its context based on a text prompt. A core challenge in this task is the "similarity-controllability paradox", where enhancing textual control often degrades the subject's fidelity, and vice-versa. We argue this paradox stems from the ambiguous role of text prompts, which are often tasked with describing both the subject and the desired modifications, leading to conflicting signals for the model. To resolve this, we propose DisCo, a novel framework that first Disntangles and then re-Couples visual and textual information. First, our textual-visual decoupling module isolates the sources of information: subject identity is extracted exclusively from the reference image with the entity word of the subject, while the text prompt is simplified to contain only the modification command, where the subject refers to general pronouns, eliminating descriptive ambiguity. However, this strict separation can lead to unnatural compositions between the subject and its contexts. We address this by designing a dedicated reward signal and using reinforcement learning to seamlessly recouple the visually-defined subject and the textually-generated context. Our approach effectively resolves the paradox, enabling simultaneous high-fidelity subject preservation and precise textual control. Extensive experiments demonstrate that our method achieves state-of-the-art performance, producing highly realistic and coherent images.
