Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To resolve the representational bottleneck where fixed-capacity recurrent states in Gated Linear Attention (GLA) must simultaneously track local syntax and global semantics at a single token resolution, the authors develop Multi-Scale Gated Linear Attention (MS-GLA). The architecture distributes attention heads across multiple temporal scales via token pooling—allocating coarse heads to long-range dependencies and fine heads to local context—before recombining them with an input-dependent fusion layer. At matched parameter counts, MS-GLA outperforms vanilla GLA across language modeling and retrieval benchmarks, achieving up to a 9.5% perplexity reduction and an 18.9% gain on recall-intensive tasks.
Linear attention models do not need larger hidden states to fix their memory bottleneck; simply assigning attention heads to distinct temporal resolutions unlocks up to an 18.9% boost on associative recall.
Gated Linear Attention (GLA) Transformers advance linear recurrent models through data-dependent gating, but face a core limitation: the fixed-capacity memory matrices across all heads operate at a single temporal resolution, where each token is processed individually, forcing them to simultaneously encode local syntactic patterns and long-range semantic structure, creating a representational bottleneck that gating alone is insufficient to resolve. We introduce Multi-Scale Gated Linear Attention (MS-GLA), which addresses this by distributing attention heads across multiple temporal resolutions. Coarser resolutions pool longer token spans naturally specializing toward long-range dependencies, while finer head groups retain sensitivity to local syntactic structure. A learnable, input-dependent fusion layer dynamically recombines head group outputs at each timestep, expanding effective memory capacity without increasing per-head state size. This multi-resolution decomposition draws on principles from Multi-Scale State-Space Models (MS-SSM), adapting them to the gated linear attention setting. We evaluate MS-GLA on language modeling, recall-intensive tasks, and long-context generalization. Across all settings, MS-GLA consistently achieves higher accuracy and lower perplexity than GLA at matched parameter counts, with up to 18.9% improvement on recall-intensive tasks and 9.5% lower average perplexity on language modeling benchmarks, validating multi-temporal resolution decomposition as a principled and effective extension of Gated Linear Attention.