Search papers, labs, and topics across Lattice.
This study empirically evaluates the impact of quantization on language models (LLMs) in the context of Automated Program Repair (APR), analyzing 13 configurations across six LLMs. The results indicate that while quantization can reduce memory usage by up to 85%, it also leads to increased inference time and energy consumption, revealing a trade-off that is sensitive to model architecture and task complexity. Notably, the overlap in repaired problems between base and quantized models is minimal, suggesting that quantization may alter model behavior in significant ways.
Quantization can slash memory usage by 85%, but it may also double inference time and energy costs, complicating the trade-offs in LLM deployment for software repair tasks.
Language Models (LLMs) are powerful toolsand have been increasingly adopted for complex software engineering tasks. As the number of parameters increases, results can often be improved, but this also imposes substantialmemory requirements. While quantization effectively reduces thememory footprint, its overall impact is often summarized onlyby benchmark scores, which mask changes in model behaviorand non-functional overheads. In this work, we conduct anempirical evaluation of LLM quantization using AutomatedProgram Repair (APR), a complex task in software engineering.We analyze 13 quantization configurations spanning differentbit-widths, methods, and target components (weights and KVcache) across six representative LLMs, evaluated on two APRbenchmarks (HumanEval-Java and Defects4J). Our findings reveal that base and quantized models can provide different sets of repaired problems with little overlap, whileretaining a comparable number of repaired problems. Althoughquantization successfully reduces memory footprints by up to85%, it increases both inference time and energy consumption,which we attribute to suboptimal hardware utilization. OurPareto trade-off analysis shows that 48% of the configurationsevaluated are strictly dominated by alternatives. Rather thanidentifying a superior quantization method, our findings highlightthat the trade-offs between effectiveness, memory footprint,and energy efficiency are sensitive to the underlying modelarchitecture and the complexity of the task.
