Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the heterogeneity of cross-attention maps in text-to-image diffusion models, finding that different attention heads capture concept-specific features with varying accuracy. They introduce a selective aggregation method that combines attention maps from the most relevant heads for a given concept, improving visual interpretability compared to the DAAM baseline as measured by mean IoU. The study also demonstrates that this selective aggregation can diagnose prompt misinterpretations, highlighting the potential for improved controllability.
Not all attention heads are created equal: selectively aggregating the *right* ones dramatically sharpens the interpretability of text-to-image diffusion models.
Numerous studies on text-to-image (T2I) generative models have utilized cross-attention maps to boost application performance and interpret model behavior. However, the distinct characteristics of attention maps from different attention heads remain relatively underexplored. In this study, we show that selectively aggregating cross-attention maps from heads most relevant to a target concept can improve visual interpretability. Compared to the diffusion-based segmentation method DAAM, our approach achieves higher mean IoU scores. We also find that the most relevant heads capture concept-specific features more accurately than the least relevant ones, and that selective aggregation helps diagnose prompt misinterpretations. These findings suggest that attention head selection offers a promising direction for improving the interpretability and controllability of T2I generation.