Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To uncover the inductive biases of realistic Transformers beyond trivial clustering or rank-collapse regimes, the authors apply pattern-formation theory to model forward-pass dynamics incorporating multi-head attention, positional encodings, and value-output geometry. They reveal that full architectures selectively amplify and stabilize a rich spectrum of previously unreported token dynamics, including traveling and rotating waves, whose competition and coexistence are governed by specific architectural components. Exploiting these emergent dynamics as controllable priors via task-aligned positional encodings and weight initializations substantially boosts sample efficiency and convergence speed on synthetic sequence tasks and ConViT on CIFAR-10.
Far from merely clustering tokens, full Transformers act as pattern-forming dynamical systems that natively amplify traveling and rotating waves—an inductive bias that can be engineered at initialization to accelerate training.
What are the inductive biases of a Transformer architecture? Existing theory on how the forward pass shapes representations either considers whether Transformers escape from rank collapse or demonstrates that self-attention drives tokens toward cluster patterns. The latter view arises from an elegant dynamical systems perspective, but relies on simplified architectural assumptions, and does not explain the rich structures observed in practice. This leaves a major open question: when a full Transformer escapes rank collapse, how does it structure token representations? Using pattern-formation theory, we show that the dynamical view of Transformers can account for Positional Encoding, Multi-Head Attention, and Output-Value geometry. We demonstrate that a full Transformer architecture imposes an inductive prior by selectively amplifying a rich set of previously unreported patterns, including traveling or rotating waves among others. We characterize the role of each architectural component in controlling which pattern is amplified, which ones stabilize, compete, or coexist. Finally, we show that these structures can act as a controllable dynamical prior that facilitates learning. By choosing both task-aligned positional encoding and weight initialization, we demonstrate improved data efficiency and accelerated optimization on controlled sequence tasks and with ConViT on CIFAR-10.