Search papers, labs, and topics across Lattice.
This study empirically evaluates six state-of-the-art quantization methods for large code models, specifically focusing on their impact on functional correctness and code quality. Using the multilingual McEval and CoderEval benchmarks, the authors find that quantization techniques significantly affect the generated code, with AQLM consistently matching or exceeding the full-precision baseline, while QuIP# shows the most degradation in correctness, especially with complex prompts. The research emphasizes the need to assess quantized models on multiple dimensions, including maintainability and robustness, to guide developers in selecting appropriate quantization strategies for local inference on constrained hardware.
AQLM outperforms full-precision baselines in code generation, while QuIP# falters under complex prompts, revealing critical trade-offs in quantization strategies.
The growing adoption of local inference frameworks such as Ollama has made it increasingly common for developers to run large code models on laptops and other resource-constrained hardware. In these settings, post-training quantization is essential for reducing memory footprint and enabling practical deployment, yet its impact on generated code remains insufficiently understood. We empirically evaluate six state-of-the-art quantization methods (GPTQ, AWQ, QuIP#, AQLM, BitsAndBytes, and GGUF) on two representative large code model families, Qwen2.5-Coder and CodeLlama, using the multilingual McEval and CoderEval benchmarks for Python and Java. We assess functional correctness (pass@1) together with maintainability, reliability, security, and structural complexity. We also introduce a novel analysis of robustness under varying prompt complexity, characterized by Shannon entropy and token length. Our results show that quantization techniques differ meaningfully in their impact on correctness and code quality. AQLM consistently matches or exceeds the full-precision baseline, whereas QuIP# exhibits the largest correctness degradation, particularly on complex prompts. Security attributes remain stable across models, benchmarks, and programming languages, while robustness to prompt complexity varies across techniques. These findings provide practical guidance for selecting quantization strategies for deploying large code models on resource-constrained hardware and highlight the importance of evaluating quantized models beyond functional correctness.
