Search papers, labs, and topics across Lattice.
This paper investigates the inference-time dynamics of deep linear transformers by modeling tokens as interacting particles within a self-attention framework. By reformulating the dynamics as a generalized Kuramoto-type model, the authors reveal that the behavior of these models can exhibit diverse long-term patterns such as clustering, oscillations, and bifurcations, which are tied to a hidden Hamiltonian structure. The findings indicate that even in higher-dimensional settings, the observed dynamical behaviors remain consistent, highlighting the low-dimensional nature of these transformer models' dynamics.
Long-time behaviors in deep linear transformers can reveal surprising dynamics like clustering and oscillations, all rooted in a hidden Hamiltonian structure.
We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles that interact dynamically through successive linear self-attention layers. We show that in embedding dimension two, for any key, query, and value matrices, the dynamics can be reformulated as a generalized Kuramoto-type model with pure second-harmonic coupling. This formulation is amenable to Watanabe--Strogatz theory which reveals the dynamics are intrinsically low-dimensional regardless of the parameter matrices. For a class of token initializations associated with the Ott--Antonsen (OA) manifold, we show that the parameter matrices induce a diverse variety of long-time behaviors in linear transformers, including clustering, oscillations, and bifurcations. The oscillations and bifurcations are characterized by uncovering a hidden Hamiltonian structure in the dynamics. By establishing a structural stability result, we further show that dynamics initialized near the OA manifold exhibit the same long-time behavior as those initialized exactly on the manifold. Motivated by our theory in dimension two, we conduct numerical experiments for analogous parameter regimes in higher-dimensional transformers. Our numerical experiments suggest that the long-time behaviors characterized in our theoretical results persist in higher dimensions.
