Search papers, labs, and topics across Lattice.
This paper investigates how RoPE-trained transformers manage to discern absolute positions in their attention mechanisms, despite only encoding relative offsets. The authors identify two architectural elements responsible for this phenomenon: the causal mask, which inherently ties the softmax denominator to absolute query positions, and the residual stream, which creates a closed dynamical system that influences downstream attention. Their findings reveal that modifying the \texttt{BOS} embedding can significantly reduce the influence of the residual stream, highlighting the intricate interplay between these components across different transformer architectures.
RoPE-trained transformers can leak absolute positional information through their attention mechanisms, revealing hidden architectural dependencies that could reshape model design.
RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product. We trace this leakage to two architectural components, The causal mask is responsible for the first: its per-query softmax denominator depends on the absolute query position by construction. The residual stream supplies the second. Under causal attention the activation at position $0$ attends only to itself and runs as a closed dynamical system from the embedding of the token at that position; downstream attention reads this trajectory through sink-reading heads. Both components appear in all three architectures we study, in architecturally specific balance: NTK scaling suppresses the residual-stream component, sliding-window attention allows it to accumulate with depth, and standard RoPE sits between. Replacing the \texttt{BOS} embedding before the forward pass removes $40\%$ of the residual-stream component at early queries. Attention sinks are token-anchored stabilizers that pass forward a deterministic fingerprint of the token at position $0$, constant across inputs when that token is the auto-prepended \texttt{BOS} and varying with it otherwise.
