Search papers, labs, and topics across Lattice.
This paper investigates the dynamics of self-attention mechanisms enhanced by Rotary Position Embeddings (RoPE), focusing on the effects of position-dependent rotations on token interactions within a spherical model. The authors derive a reversible attention kernel that exhibits a uniform softmax floor and analyze the implications of energy derivatives on consensus states, revealing complex behaviors in multi-dimensional settings. Key findings include the identification of a non-hyperbolic resonant family and the demonstration that energy allocation across frequency planes can lead to non-monotonic consensus gaps, challenging existing assumptions about frequency ordering in attention dynamics.
The study reveals that energy distribution across frequency planes can create unexpected consensus dynamics, defying traditional assumptions in self-attention models.
Rotary position embeddings (RoPE) modify attention scores through position-dependent rotations, but their effect on normalized token dynamics is not captured by the vanilla spherical self-attention model. We study the continuous-time dynamics obtained when queries and keys are rotated while values remain on the unit sphere. The resulting attention kernel is reversible and admits a sharp uniform softmax floor, yet the natural RoPE interaction energy has derivatives of both signs within one fixed nontrivial system. Every consensus state remains an equilibrium, and its transverse linearization is a reversible Markov operator whose kernel depends on the consensus point through its energy across RoPE planes. On a resonant single-frequency ring we derive an exact Bessel-aliasing spectrum, including non-coprime frequencies and the correct fixed-ring large-$尾$ asymptotics. Globally, closed hemispheres are invariant, while pairwise non-obtuse configurations and strict open semicircles contract with explicit half-angle and single-point tail bounds. These regional estimates instantiate a kernel-generic positivity principle with the sharp RoPE softmax floor. RoPE also selects an explicit score-flattening twisted branch; the generic resonant family is non-hyperbolic and linearly unstable, whereas an odd antipodal family becomes a hyperbolic saddle after quotienting global rotation. In multiple dimensions, the local consensus gap can depend non-monotonically on the allocation of energy across frequency planes, so no universal ordering by frequency is valid. Independent matrix, finite-difference, and nonlinear-flow computations cross-check the theorem boundaries and the reported constants.
