Search papers, labs, and topics across Lattice.
This study investigates the concept of functional equivalence in Transformers, particularly focusing on the impact of positional encodings鈥攕pecifically sinusoidal and rotary positional encodings (RoPE)鈥攐n the symmetry of attention mechanisms. The authors find that while sinusoidal encodings maintain the equivalence structure of traditional attention, RoPE significantly reduces the symmetry group, which enhances the model's expressivity. Additionally, they demonstrate that the choice of positional encoding influences linear mode connectivity, revealing a critical relationship between encoding types and connectivity variability in Transformer architectures.
Rotary positional encodings not only enhance expressivity in Transformers but also fundamentally alter the symmetry landscape of attention mechanisms.
Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence. While this symmetry is well understood in classical fully connected and convolutional models, it becomes substantially more intricate in modern attention-based architectures. Existing analyses of multihead attention have largely focused on the vanilla formulation, overlooking positional encodings that fundamentally reshape architectural symmetries. In this work, we provide a formal study of functional equivalence in Transformers with positional encodings. Focusing on the two most widely used variants--sinusoidal and rotary positional encodings (RoPE)--we show that sinusoidal encodings preserve the equivalence structure of vanilla attention, whereas rotary encodings significantly reduce the symmetry group, thereby enhancing expressivity. This offers a principled explanation for the growing prominence of RoPE in practice. We further examine how positional encodings affect linear mode connectivity, and through an alignment algorithm, empirically demonstrate that the presence and variability of connectivity across Transformer settings crucially depend on the positional encoding.
