Search papers, labs, and topics across Lattice.
This study reveals that seemingly minor architectural choices in dense transformer models significantly affect their ability to handle long contexts, with a combination of specific decisions leading to performance drops of up to 47%. By conducting controlled ablations while fixing data and tokenizer variables, the authors identify that variations in normalization, GQA, pretraining context length, and sliding window attention are critical in determining long context extensibility. The findings underscore that these architectural features, which are not detectable through short-context evaluations, drive substantial differences in performance across model families, leading to the release of the OlmPool set of 26 models that excel in long context tasks.
Minor architectural tweaks can lead to a staggering 47% drop in long context performance, challenging assumptions about model design.
One might imagine that architectural variations within the dense transformer paradigm have a limited effect on accuracy. However, we demonstrate that this is not the case in the long context setting. Specifically, we show that a set of four minor architectural decisions --- all made by at least one of the Olmo, Llama, and Qwen dense model families --- have a compoundingly negative effect on long context extensibility. Any one of these choices alone has a minor impact on long context performance, but combining three or more can drop the performance downstream by up to 47%. Furthermore, these differences are not detectable from short-context loss or validation datasets. We show that much of the variation in long context ability across model families is driven by these architectural features and detectable from applying context extension early in pretraining. We demonstrate this with controlled ablations that hold data, tokenizer, and extension recipe fixed while varying normalization, GQA, pretraining context length, and sliding window attention. After over 170,000 GPU hours of training, we release the resulting set of models as OlmPool, a set of 26 comparable 7B models with checkpoints before and after long-context extension. This pool includes several architectures that outperform the Llama 3 architecture on long context extensibility. In an analysis of our ablation models, we identify patterns in attention sink behavior and attention distributions across context that are attributable to specific architectural differences.
CMU ML