Search papers, labs, and topics across Lattice.
This paper reveals that massive activation spikes in Large Language Models (LLMs) are not merely high-level scalar biases but are instead structural vector biases linked to spike-carrying tokens. The authors demonstrate that these tokens stabilize into constant vectors that influence attention and value-state mechanisms, while also resisting perturbations from Rotary Positional Embedding (RoPE). They introduce INSERTQUANT, a post-training quantization framework that effectively eliminates spikes, allowing for robust low-bit quantization without sacrificing fidelity, achieving results comparable to leading quantization methods across various modalities.
Massive activation spikes in LLMs are structural biases that can be eliminated to enable high-fidelity low-bit quantization across modalities.
Massive activation spikes in Large Language Models (LLMs) severely degrade quantization by stretching dynamic ranges. While prior hypotheses characterize these as high-level scalar biases, we argue that they are merely the scalar intermediates of rigid, structural vector biases in the spike-carrying tokens. We show that these tokens converge to constant vectors after normalization that drive the attention sink and value-state drain mechanisms. We geometrically substantiate this by analyzing the coordination of projection weights: $W_K$ contrastively amplifies the vector, $W_Q$ aligns semantic tokens toward it, and $W_V$ projects it into the spectral null-space. Furthermore, we reveal that the model actively preserves these structural biases against Rotary Positional Embedding (RoPE) perturbations by localizing them in "zones of rotational stability" utilizing low-frequency bands and coherent channel pairs. Leveraging this, we propose INSERTQUANT, a post-training quantization (PTQ) framework that clamps spikes and restores their function via pre-computed template vectors. This renders activations strictly spike-free, enabling robust low-bit quantization with high fidelity. INSERTQUANT achieves parity with state-of-the-art per-tensor quantization methods on LLMs and uniquely generalizes beyond text to other modalities such as ViTs.
