Search papers, labs, and topics across Lattice.
This paper introduces a theoretical framework for understanding deep transformers as mean-field interacting systems that utilize distributed inference under constraints of communication, locality, and depth. The authors reveal that transformers can leverage internal state representations, termed 'function vectors', to progressively infer latent context variables at finer scales across layers. Empirical tests on constrained linear attention transformers validate the theory, showing that transformer depth significantly influences in-context learning capabilities, enabling a richer class of algorithms than previously recognized.
Transformers can adaptively infer complex latent contexts, revealing a surprising link between model depth and in-context learning efficiency.
Transformers are widely used as a general-purpose substrate for learning complex correlations between a large collection of coupled variables, but their internal mechanisms have remained mysterious. We introduce a theory of a deep transformer as a mean-field interacting system that implements distributed inference, subject to constraints on communication, locality and depth. We show that such a system can exploit internal state representations ('function vectors') to infer a latent context variable at increasingly finer scales over its layers. In an in-context regression task, the theory predicts a non-trivial relationship between non-Gaussian, hierarchical structure in the latent context variable, and transformer depth. Predictions are tested using constrained linear attention transformers and demonstrate adaptive inference in deep architectures. Feedforward blocks and depth enable transformers to implement a much richer class of in-context learning algorithms than previously described.
