Search papers, labs, and topics across Lattice.
This paper introduces Training-aware Low-Rank Adaptation Initialization (TaRA), a novel method that enhances the initialization of Low-Rank Adaptation (LoRA) by aligning the gradients of low-rank factors with those of the full-rank weight matrix. By addressing the initialization sensitivity inherent in LoRA, TaRA significantly improves gradient fidelity at the start of training while maintaining low computational overhead. Experimental results demonstrate that TaRA consistently outperforms existing state-of-the-art initialization methods across a variety of fine-tuning tasks, establishing it as a robust solution for parameter-efficient fine-tuning.
TaRA achieves superior initialization for LoRA by ensuring low-rank gradients closely match full-rank gradients, leading to better performance in fine-tuning tasks.
Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.
