Search papers, labs, and topics across Lattice.
This paper introduces ZO-Act, a novel zeroth-order optimization method for fine-tuning large language models that leverages activation-informed low-rank subspaces to enhance performance while minimizing variance. By restricting perturbations to a fixed low-rank subspace derived from input activations and optimizing lightweight coefficient matrices, ZO-Act achieves more stable convergence and supports quantized model fine-tuning. Experimental results demonstrate significant improvements in language understanding, question answering, and commonsense reasoning tasks compared to existing zeroth-order fine-tuning techniques.
Fine-tuning large language models with ZO-Act yields consistent performance gains while dramatically reducing variance in optimization.
Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance. We propose ZO-Act, an activation-informed ZO fine-tuning method that restricts perturbations to a fixed low-rank subspace derived from input activations. For each linear layer, ZO-Act computes a small activation basis once at initialization and optimizes only lightweight coefficient matrices using forward-only loss evaluations. This reduces the effective perturbation dimension, exposes explicit trainable variables compatible with momentum-based optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping low-bit weights frozen. We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space and show that perturbing the low-dimensional coefficients reduces both the variance-dependent convergence term and the finite-difference error of the ZO estimator, at the cost of a controlled subspace approximation bias that is mitigated by the low-rank structure of LLM activations and gradients. Experiments on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B show consistent gains over strong ZO fine-tuning baselines across language understanding, question answering, and commonsense reasoning.
