Search papers, labs, and topics across Lattice.
The paper introduces Quantized Evolution Strategies (QES), a novel optimization paradigm for fine-tuning quantized Large Language Models (LLMs) directly in the quantized space, addressing the limitations of backpropagation-based methods and standard Evolution Strategies (ES). QES incorporates accumulated error feedback to preserve high-precision gradient signals and employs stateless seed replay to reduce memory usage to low-precision inference levels. Experiments on arithmetic reasoning tasks demonstrate that QES significantly outperforms existing zeroth-order fine-tuning methods, enabling effective fine-tuning of quantized LLMs.
Fine-tune quantized LLMs without backpropagation using QES, a method that achieves high-precision results at low-precision cost.
Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune. Standard fine-tuning paradigms, including Reinforcement Learning (RL), fundamentally rely on backpropagation and high-precision weights to compute gradients. Thus they cannot be used on quantized models, where the parameter space is discrete and non-differentiable. While Evolution Strategies (ES) offer a backpropagation-free alternative, optimization of the quantized parameters can still fail due to vanishing or inaccurate gradient. This paper introduces Quantized Evolution Strategies (QES), an optimization paradigm that performs full-parameter fine-tuning directly in the quantized space. QES is based on two innovations: (1) it integrates accumulated error feedback to preserve high-precision gradient signals, and (2) it utilizes a stateless seed replay to reduce memory usage to low-precision inference levels. QES significantly outperforms the state-of-the-art zeroth-order fine-tuning method on arithmetic reasoning tasks, making direct fine-tuning for quantized models possible. It therefore opens up the possibility for scaling up LLMs entirely in the quantized space. The source code is available at https://github.com/dibbla/Quantized-Evolution-Strategies .
