Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides the first comprehensive evaluation of weight-only quantization strategies for LLM inference, quantifying their impact on energy usage and carbon intensity across various models, datasets, and GPUs. It reveals that quantization techniques achieve peak energy efficiency with long inputs and short outputs, and that fused-kernel implementations like EETQ int8 and Bitsandbytes int8 can yield up to 4x energy savings for short text generation with minimal accuracy loss. The study concludes that runtime closely correlates with energy usage, suggesting latency optimization as a practical proxy for reducing LLM environmental footprints.
Quantization strategies can achieve up to 4x energy savings for LLM inference, with peak efficiency observed for long inputs and short outputs.
While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting impact on energy usage and carbon intensity are scarce. We present the first unified evaluation of weight-only quantization strategies for LLM inference across varying input/output lengths and GPUs, highlighting differences in energy efficiency in addition to accuracy degradation and runtime. We study quantized inference on Llama 2 7B, Phi 3.5 Mini Instruct, OLMo 1B, Qwen 2.5 7B, and Qwen 2.5 14B across GLUE MNLI, MMLU, HumanEval, and GSM8K datasets, evaluating 10 post-training quantization (PTQ) strategies on NVIDIA H100, A6000, A100, and L4 GPUs. We identify a novel trend showing that quantization techniques tend to exhibit peak energy efficiency relative to full-precision baselines when inputs are sufficiently long and outputs are short. Furthermore, we show that quantization strategies become marginally more energy-efficient relative to full-precision models as batch sizes increase, though gains are modest. Notably, fused-kernel implementations such as EETQ int8 and Bitsandbytes int8 offer the highest energy savings, up to 4x compared to FP32 on short text generation, with negligible accuracy loss. Finally, we observe that energy usage closely tracks runtime on our evaluated benchmarks, indicating that, in practice, latency optimization can serve as a reliable proxy for reducing the environmental footprint of LLM services. We conclude with suggested directions for strategically selecting low-carbon quantization strategies based on specific inference requirements.