Search papers, labs, and topics across Lattice.
This paper introduces AI-T2I, a novel approach to improve text-to-image alignment in diffusion models by addressing both inter-subject-token overlap and intra-subject-token scattering in cross-attention maps. They propose an aggregation loss to consolidate scattered intra-token activations and an isolation loss to separate inter-token activations. Experiments demonstrate AI-T2I's superior performance on text-to-image synthesis benchmarks and its generalization to controllable layout and personalized generation tasks.
Sharper text-to-image alignment is now possible in diffusion models by explicitly aggregating related attention and isolating unrelated attention.
Text-to-image synthesis has made significant progress, benefiting from the strong generative capabilities of diffusion models. However, these models struggle to achieve precise text-to-image alignment within cross-attention maps during the denoising process. Existing works primarily focus on inter-subject-token activations (i.e., cross-attention scores) overlap for different subjects, overlooking the intra-subject-token activations scattering issue for identical subjects. In this paper, we propose an Aggregating-and-Isolating cross-attention approach to diffusion models for Text-to-Image synthesis, dubbed AI-T2I. Technically, to address the scattering issue, we devise an aggregation loss to identify and consolidate the scattered intra-token activations, which implicitly helps mitigate the potential overlap issue. Upon that, an isolation loss is further introduced to push the inter-token activations apart, thus fulfilling precise text-to-image alignment. Extensive experiments on various benchmarks demonstrate the superiority of AI-T2I over the state-of-the-art works for text-to-image synthesis. Furthermore, our AI-T2I exhibits excellent generalization across other tasks, e.g., controllable layout generation and personalized generation.
