Search papers, labs, and topics across Lattice.
This paper analyzes the attention matrix in diffusion model transformers by decomposing it into symmetric and skew-symmetric components, interpreting the former as defining an energy landscape and the latter as driving circulation on it. They derive Hopfield-style stability measures from the symmetric component to quantify the stability of retrieved features and correlate these measures with the fidelity-diversity trade-off in generated images. The authors then introduce a method to control this trade-off by modulating the circulation induced by the skew-symmetric component.
Symmetric attention is the key to balancing fidelity and diversity in diffusion models, offering a new control knob for image generation.
We characterize the pre-softmax attention matrix $\mathbf{QK^\top}$ in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric parts, we interpret the symmetric component as governing the structure of the energy landscape, and the skew-symmetric component as driving circulation on that landscape. Leveraging the energy formulation induced by the symmetric component, we derive Hopfield-style stability measures that quantify the stability of retrieved features. We observe meaningful correlations between Hopfield-style stability measures and the fidelity-diversity trade-offs in generation. Finally, we propose a controllable knob to modulate this trade-off by modifying the circulation of the underlying dynamics. Code is available at our GitHub (https://github.com/hyeon-cho/Attention-Symmetric-Decomposition).
