Search papers, labs, and topics across Lattice.
This paper introduces LoRA-GA$^2$, an advanced fine-tuning algorithm that enhances Low-Rank Adaptation (LoRA) by utilizing multi-step gradient information to align updates more effectively with the dynamics of full fine-tuning. By implementing a lightweight probe for capturing these gradients without incurring additional GPU memory costs, the method achieves significant performance improvements over existing LoRA variants while maintaining efficiency. Experimental results reveal that LoRA-GA$^2$ outperforms leading baselines on multiple benchmarks, including an average improvement of 0.66 points on GLUE and 1.03 points on GSM8K.
LoRA-GA$^2$ closes the performance gap with full fine-tuning by leveraging multi-step gradient dynamics without sacrificing efficiency.
Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead. However, a persistent performance gap remains between LoRA and full fine-tuning. Recent studies have sought to narrow this gap by employing one-step gradient approximations of pretrained weights to align LoRA updates with the principal directions or intrinsic dimensionalities of full fine-tuning updates. Nevertheless, these approaches fail to capture the full dynamics of the gradients. In this paper, we propose LoRA-GA$^2$, an effective fine-tuning algorithm that fully leverages multi-step gradient information. Specifically, we introduce a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead. We further employ a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients. Extensive experimental results demonstrate that LoRA-GA$^2$ consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA. For instance, LoRA-GA$^2$ surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark, and outperforms the strongest baseline by 1.03 points on GSM8K and 0.87 points on HumanEval, respectively.
