Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a theoretical analysis of positional encodings (PEs) in Transformers, demonstrating their necessity for order-sensitive tasks and proving that training leads to distinct representations for each position. They show that the optimal PE can be constructed using multidimensional scaling (MDS) on Hellinger distances between positional distributions, and its quality can be quantified by "stress." Empirically, they find that ALiBi achieves lower stress than sinusoidal and RoPE encodings, aligning with a low-rank interpretation of the MDS encoding.
Forget trial-and-error: this paper mathematically derives the *optimal* positional encoding for Transformers using multidimensional scaling, revealing that ALiBi is surprisingly close.
Neural language models process sequences of words, but the mathematical operations inside them are insensitive to the order in which words appear. Positional encodings are the component added to remedy this. Despite their importance, positional encodings have been designed largely by trial and error, without a mathematical theory of what they ought to do. This paper develops such a theory. Four results are established. First, any Transformer without a positional signal cannot solve any task sensitive to word order (Necessity Theorem). Second, training assigns distinct vector representations to distinct sequence positions at every global minimiser, under mild and verifiable conditions (Positional Separation Theorem). Third, the best achievable approximation to an information-optimal encoding is constructed via classical multidimensional scaling (MDS) on the Hellinger distance between positional distributions; the quality of any encoding is measured by a single number, the stress (Proposition 5, Algorithm 1). Fourth, the optimal encoding has effective rank r = rank(B)<= n-1 and can be represented with r(n+d) parameters instead of nd (minimal parametrisation result). Appendix A develops a proof of the Monotonicity Conjecture within the Neural Tangent Kernel (NTK) regime for masked language modelling (MLM) losses, sequence classification losses, and general losses satisfying a positional sufficiency condition, through five lemmas. Experiments on SST-2 and IMDB with BERT-base confirm the theoretical predictions and reveal that Attention with Linear Biases (ALiBi) achieves much lower stress than the sinusoidal encoding and Rotary Position Embedding (RoPE), consistent with a rank-1 interpretation of the MDS encoding under approximate shift-equivariance.