Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper analyzes the discrete-time dynamics of the self-attention mechanism in Transformers, treating layer index as time. It provides a complete classification of equilibria and their stability, revealing that the primary attractor is a consensus equilibrium aligned with the principal eigenvector of the value matrix. This theoretical analysis is validated by observing learned weight matrices in a real Transformer.
Discrete-time self-attention dynamics converge to a consensus equilibrium dictated by the principal eigenvector of the value matrix, a finding confirmed in real-world Transformers.
In machine learning, a self-attention dynamics describes the attention mechanism of transformers, and is obtained treating the layer index of a transformer as time index. Even though naturally discrete-time, this dynamical system is most often studied in continuous-time. Scope of this paper is to provide a thorough investigation of the discrete-time version of the self-attention dynamics: a complete classification of its equilibria is given, and their stability properties are studied, in both the single-head and the multihead case. We show that in both cases the main attractor of the dynamics is a consensus equilibrium aligned with the principal eigenvector of the value matrix (in the single-head case, a corresponding matrix of weights in the multihead case), even though other locally asymptotically stable bipartite equilibria can be show to exist at least in the single-head situation. The behavior is confirmed by looking at the learned weight matrices of a real transformer.