Search papers, labs, and topics across Lattice.
This paper investigates the differing generalization capabilities of transformers with relative versus absolute positional encodings, revealing that relative encodings enable extrapolation to longer sequences due to their inherent optimization properties. The authors demonstrate that rotary encodings ensure the attention logit is solely a function of relative offsets, allowing learned selection rules to be consistently applied beyond training lengths. In contrast, absolute encodings constrain attention to fixed positions, leading to poor generalization; this is characterized through a low-rank carrier kernel and confirmed via empirical validation across various settings.
Relative positional encodings not only enable extrapolation in transformers but also reveal a profound connection between implicit bias and sequence length generalization.
Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not. This is a robust empirical regularity, and the explanations offered for it so far are chiefly about expressivity, that is, about whether a length-generalizing solution exists. We give an optimization explanation. On a minimal fixed-offset retrieval task that isolates positional selection, the gap is governed by the implicit bias of the trained attention head: among the many solutions that fit short sequences, which one gradient descent actually selects. We prove that rotary encodings make the attention logit a function of relative offset alone, an exact equivariance, so whatever selection rule is learned at training lengths is reproduced verbatim at every longer length. Learned absolute encodings instead leave out-of-range positions unconstrained, and the trained head pins to a fixed absolute position inside the training range. We characterize the learned rotary rule as a low-rank ``carrier''kernel aligned with the target offset, and we derive the resulting graceful accuracy decay as an attention-dilution law; both predictions are confirmed across seeds and offsets. A linear-attention control shows the mechanism is specific to softmax: without normalization, training selects a min-norm interpolant that does not extrapolate. The phenomenon, the equivariance, and the carrier all transfer to a multi-layer, multi-head transformer trained on a full-sequence length-generalization task. The account connects the implicit bias of attention, implicit bias for extrapolation in recurrent models, and the learning side of the RASP-L conjecture.
