Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces GaugeQuant, a novel approach that optimizes quantization in transformers by leveraging internal continuous symmetries during training. By incorporating a LogSumExp term into the loss function and utilizing a stop-gradient operator, GaugeQuant effectively minimizes activation outliers while maintaining the integrity of the language modeling objective. The method demonstrates significant improvements in perplexity for the LLaMA-2 7B model under various quantization settings, outperforming traditional post-training quantization methods without the need for calibration data or extensive computational overhead.
Achieving a perplexity reduction from 8.22 to 6.73 under W4A4 quantization, GaugeQuant rivals conventional post-training methods without requiring frozen models or calibration datasets.
Transformers are known to have internal continuous symmetries that leave outputs invariant, while modifying quantization. GaugeQuant leverages this in-training by introducing a LogSumExp term to the loss that breaks the symmetries, thus selecting a basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, yielding the language modeling objective completely unaltered. Our requires no specific calibration data, no quantization simulation, and adds negligible training overhead. With the LLaMA-2 7B model under W4A4 quantization with group size 128, perplexity drops from 8.22 to 6.73, competing with post-training methods that require frozen models and calibration datasets. Under W4A16, perplexity drops from 11.16 to 5.45. Code is available at https://github.com/MPedraBento/gauge-quant.