Search papers, labs, and topics across Lattice.
The paper investigates the impact of residual connections on the quantization robustness of transformers, finding that they exacerbate non-Gaussian activation distributions, leading to higher quantization error. They demonstrate that residual-free transformers, when trained with techniques like orthogonal initialization and depth-aware scaling, exhibit activations more amenable to quantization. Empirically, residual-free transformers show improved robustness to low-bit quantization with only a small drop in full-precision performance, highlighting a trade-off between accuracy and compressibility in transformer architecture design.
Residual connections, a staple of modern transformers, actively *harm* quantization robustness by driving activations away from Gaussianity.
Large-scale transformer training and deployment are increasingly constrained by the transfer of activations, gradients, and optimizer states across accelerators. Low-bit quantization offers a natural remedy, but transformer activations are often heavy-tailed and outlier-dominated, making simple quantization highly lossy. We show that this difficulty is not only a property of the quantizer, but also of the architecture. Specifically, residual connections can drive transformer activations away from Gaussianity during training. Using controlled comparisons between residual and residual-free transformers, we demonstrate that this effect leads to substantially higher quantization error and accuracy degradation at low precision in residual models. We explain the phenomenon through an excess kurtosis analysis, showing that residual mixing can amplify non-Gaussianity, whereas dense mixing in residual-free contracts non-Gaussianity. We then show that residual-free transformers can be made trainable using orthogonal initialization, spectral or second-order optimization, and depth-aware scaling of attention temperature. In language tasks, while there is a small drop in full precision performance, these models retain near-Gaussian activations and exhibit significantly improved robustness to low-bit quantization. Our results identify an accuracy--compressibility trade-off in transformer design and motivate architecture-level approaches to quantization-friendly foundation models.
