Search papers, labs, and topics across Lattice.
This paper critiques the limitations of Low-Rank Adaptation (LoRA) by revealing how anisotropic gradient scaling affects the backpropagation of gradients through low-rank matrices, leading to suboptimal model adaptation. The authors introduce SDS-LoRA, a novel parameterization that decouples singular values from the backward pass, allowing for a more effective alignment between full fine-tuning gradients and their low-rank approximations. Experimental results across various benchmarks demonstrate that SDS-LoRA not only improves convergence rates but also significantly narrows the performance gap compared to full fine-tuning.
Anisotropic gradient scaling in LoRA can severely hinder model adaptation, but SDS-LoRA effectively mitigates this issue, leading to superior performance.
Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices. In this paper, we investigate the limitations of the LoRA parameterization from a geometric perspective. Specifically, we show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values. We argue that this phenomenon is undesirable because it distorts the full fine-tuning gradient by skewing it toward dominant singular directions while suppressing others. Our analyses demonstrate that anisotropic gradient scaling reduces the effective rank of the low-rank matrices' gradients and results in suboptimal alignment between the full fine-tuning gradient and its low-rank approximation in LoRA, thereby exacerbating the gap to full fine-tuning. To address these limitations, we propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass. Our method ensures that the full fine-tuning gradient backpropagates only through the orthonormal bases of the low-rank matrices' subspaces, independent of their scales. Convergence analysis demonstrates that while LoRA's convergence rate degrades with the condition number of the low-rank matrices, SDS-LoRA remains independent of it. Experimental results across natural language and vision benchmarks show that SDS-LoRA improves loss convergence and reduces the gap to full fine-tuning, significantly enhancing adaptation performance.
