Search papers, labs, and topics across Lattice.
The authors investigate the underlying mechanics driving attention sinks and sequence-initial massive activations, directly challenging the assumption that positional encodings like RoPE are the root cause. Their findings reveal that these phenomena stem from causal mask-induced self-concentration at the initial token and subsequent value-non-mixing in attention outputs, regardless of token identity. Pinpointing these causal mechanisms provides a clearer roadmap for addressing outlier activations that routinely degrade low-bit LLM quantization.
Contrary to widespread assumptions, RoPE is not responsible for attention sinks and outlier activations; the true culprits are causal-mask self-concentration and value-non-mixing at the sequence start.
Large Language Models (LLMs) often exhibit"Attention Sink"(AS) and the accompanying"Massive Activations"(MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it. Our experiments suggest that self-concentration of attention, resulting from the causal mask, and the subsequent Value-non-mixing in attention outputs contribute to AS and MAs. These findings provide new empirical evidence on the internal dynamics of LLMs, offering insights that may inform future quantization strategies and advance our understanding of the internal mechanisms of attention layers.
