Search papers, labs, and topics across Lattice.
This paper investigates the in-context learning capabilities of linear transformers, addressing the computational inefficiencies of traditional softmax transformers that exhibit quadratic complexity with respect to context length. By employing a two-staged sampling process from domain generalization, the authors reveal that linear transformers can effectively learn mappings from context distributions to response functions, achieving a dimension-independent convergence rate in their generalization analysis. Their findings also highlight a tradeoff between data distribution regularities and latent features, leading to a novel perspective on activation and loss design for linearized pretrained models.
Linear transformers can achieve efficient in-context learning by mapping context distributions to response functions, challenging the traditional softmax approach.
Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning. With richer context, transformers adapt more effectively to the current use case without any parameter updates. However, the quadratic computational and memory complexity with respect to context length significantly slows data processing in softmax transformers. Linear transformers were proposed to address this issue by reducing the complexity to linear dependence on context length, but the design and understanding of the feature mapping in linear attention, from a theoretical viewpoint, remain unclear. In this paper, we investigate the approximation and generalization abilities of linear transformers under a two-staged sampling process from domain generalization. We show that linear transformers perform in-context learning as learning a mapping from context distributions to response functions. A dimension-independent convergence rate is obtained for our generalization analysis, which also exhibits the tradeoff between the regularities of data distributions and latent features. Guided by our theoretical framework, we propose a new perspective on activation and loss design for linearizing pretrained softmax large language models.
