Search papers, labs, and topics across Lattice.
This paper investigates the impact of nonuniform width allocation in transformer architectures by introducing the times-shaped > <former, which features wider layers at the beginning and end while narrowing the middle layers. The results show that this architecture consistently outperforms uniform baselines in language modeling loss across various model sizes, while also achieving significant reductions in FLOPs and memory costs. By demonstrating that a tailored width allocation can optimize resource usage, this work provides a new perspective on scaling transformer models effectively.
Nonuniform width allocation in transformers can lead to a 22% reduction in FLOPs while enhancing language modeling performance.
Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across all layers, allocating a fixed parameter and computation budget evenly despite different layers potentially playing distinct computational roles. In this work, we empirically investigate nonuniform capacity allocation across network depth by proposing a times-shaped > <former architecture. This design maintains wider early and late layers while narrowing the middle layers, utilizing a parameter-free residual resizing mechanism. Across decoder-only language models ranging from 200M to 2B parameters (dense) and 3B parameters (MoE), our > <former consistently outperforms parameter-matched uniform baselines on language modeling loss. By reducing the average layer width, this architecture also requires fewer overall FLOPs (22% reduction under fitted loss-matched scaling curves) and smaller KV cache memory and I/O cost (15% reduction). In analysis, we show that this bottleneck structure results in qualitatively different representations in residual streams. Overall, our results demonstrate that nonuniform width allocation can result in more resource-optimal scaling of language models.
MIT CSAIL