Search papers, labs, and topics across Lattice.
This study formalizes two competing accounts of rotary position embeddings (RoPE) in soft-attention transformers, focusing on their expressivity and implications for language recognition. The authors demonstrate that while periodic RoPE can recognize languages defined by past temporal logic, conventional RoPE's non-repeating rotations lead to a bounded simulation of fixed-offset look-back operators, limiting its effectiveness in tasks requiring distant context access. Controlled experiments corroborate these findings, revealing that periodic schedules generalize better on modular languages compared to conventional RoPE, which exhibits a locality bias detrimental to certain tasks.
Conventional RoPE's non-repeating rotations may hinder its ability to access distant context, unlike periodic RoPE, which excels at recognizing modular languages.
Two accounts recur in explanations of the success of rotary position embeddings (RoPE). Expressivity studies associate periodic position information with modular predicates, whereas mechanistic and long-context studies emphasize positional anchors and local offsets. We formalize both accounts for fully uniform, finite-precision soft-attention transformers. We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates. Conventional RoPE is different: The rotations it computes never repeat. This yields a precision-dependent bounded simulation of fixed-offset look-back operators, rather than an all-length modular characterization. Controlled experiments match this separation: Constructed periodic schedules length-generalize on modular languages, while conventional RoPE behaves more like a bounded locality bias and can impair tasks requiring position-invariant access to distant context. Altogether, our findings shed light on RoPE transformers, bringing theoretical expressivity characterizations closer to models used in practice.
AI2