Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper characterizes equivariant multi-head self-attention (MHSA) by demonstrating that if an MHSA layer is equivariant to a symmetry group \( G \), the group can only permute head-clusters, with specific constraints on the QK and OV matrices. The authors prove that any fixed MHSA architecture achieving exact equivariance through polynomial parameterization results in expressivity loss, as the equivariance locus forms a complex union of Zariski-irreducible components in a reduced parameter space. Specifically, for the symmetry group \( G=D_4 \) with \( C \) copies of the regular representation, they show that there are \( Ω(C^{64}) \) components for eight attention heads, indicating a significant limitation in the expressivity of fixed architectures.
Fixed equivariant architectures for multi-head self-attention can lead to severe expressivity loss, with an explosion of irreducible components limiting their effectiveness.
We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For $G=D_4$ acting on $C$ copies of the regular representation as the token feature space, we show that there are $Ω(C^{64})$ components for eight attention heads.