Search papers, labs, and topics across Lattice.
This paper investigates the theoretical underpinnings of Low-Rank Adaptation (LoRA) and identifies a stability issue arising from non-zero initialization of the low-rank matrix A. To address this, they propose Stable-LoRA, a weight-shrinkage optimization strategy that dynamically enhances the stability of LoRA feature learning by progressively shrinking A during initial training steps. Empirical results demonstrate that Stable-LoRA outperforms existing baselines across various models and tasks with minimal overhead.
LoRA's feature learning isn't as stable as we thought, but a simple weight-shrinkage trick can fix it, boosting performance without extra memory.
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient method for fine-tuning Large Langauge Models. It updates the weight matrix as $W=W_0+sBA$, where $W_0$ is the original frozen weight, $s$ is a scaling factor and $A$,$B$ are trainable low-rank matrices. Despite its robust empirical effectiveness, the theoretical foundations of LoRA remain insufficiently understood, particularly with respect to feature learning stability. In this paper, we first establish that, LoRA can, in principle, naturally achieve and sustain stable feature learning (i.e., be self-stabilized) under appropriate hyper-parameters and initializations of $A$ and $B$. However, we also uncover a fundamental limitation that the necessary non-zero initialization of $A$ compromises self-stability, leading to suboptimal performances. To address this challenge, we propose Stable-LoRA, a weight-shrinkage optimization strategy that dynamically enhances stability of LoRA feature learning. By progressively shrinking $A$ during the earliest training steps, Stable-LoRA is both theoretically and empirically validated to effectively eliminate instability of LoRA feature learning while preserving the benefits of the non-zero start. Experiments show that Stable-LoRA consistently outperforms other baselines across diverse models and tasks, with no additional memory usage and only negligible computation overheads. The code is available at https://github.com/Yize-Wu/Stable-LoRA.
