Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper uncovers a recurrent algebraic structure in Transformer attention, revealing that a sparse subset of effective OV operators exhibits a near closure under composition, with 3.98–8.00% of heads achieving high alignment across various pretrained models. The authors employ a principal-coordinate factorization to differentiate between transport and return geometries, demonstrating that altering the orientation of key matrices significantly impacts closure metrics. The findings suggest that while high closure is theoretically attainable in all layers, it is often not achieved, indicating a gap between available capacity and the orientations realized by effective heads.
A surprising 3.98–8.00% of Transformer heads achieve near closure under composition, revealing a hidden algebraic structure that could redefine our understanding of attention mechanisms.
We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators $T=OV^\top$ nearly closes under composition, $T^2\approxαT$. Across six pretrained endpoints spanning 2.8B--235B parameters, 3.98--8.00% of heads reach squared closure alignment $\mathcal{P}\geq0.9$, while no matched within-layer O/V mismatch does. An exact principal-coordinate factorization, $T=Q_OKQ_V^\top$ and $T^2=Q_O(KDK)Q_V^\top$, separates within-support transport from read--write return geometry. Across all 7,304 heads in nine MHA/GQA models, scrambling only the orientation of $K$ while preserving singular values, norms, factor spans, and principal angles reduces median closure from 0.336 to $1.04\times10^{-4}$; trained orientation wins for 98.64% of heads and in every layer. Constructive searches show that high closure is feasible in every surveyed layer, but usually not attained. Retrospective trajectories in three independently trained lineages further separate broadly available capacity from the orientations attained by final strong heads. Under exact value sharing, headwise closure extends to a right-action algebra, $T_iT_j=α_jT_i$. Seven-model experiments verify the approximate law and reveal distinct oblique projections with a shared value-defined kernel. These results characterize scaled idempotence as a sparse trained orientation within broadly available geometric capacity and show how value sharing extends a headwise relation into a local operator algebra.