Search papers, labs, and topics across Lattice.
This paper uncovers a critical failure mode in ALiBi positional encoding, where linear bias scaling leads to floating-point precision underflows that significantly diminish attention weights in certain heads. Through extensive pretraining experiments on 148M-parameter decoder models, the authors demonstrate that this issue can severely hinder token retrieval while minimally affecting standard decoder performance. They propose and evaluate four mitigation strategies, identifying log-scaled distances as the most effective for improving retrieval outcomes, while still affirming the robustness of default ALiBi slopes for specific tasks.
ALiBi positional encoding can blind attention heads, drastically impairing token retrieval without impacting standard performance metrics.
We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.
