Search papers, labs, and topics across Lattice.
This paper investigates the scaling factor $\alpha$ in Low-Rank Adaptation (LoRA), revealing its critical role as a primary driver of optimization rather than a mere complement to the learning rate. Through empirical analysis and a theoretical Signal-Drift framework, the authors demonstrate that $\alpha$ enhances convergence by amplifying task signals while maintaining a low drift ratio, outperforming traditional learning rate adjustments. Additionally, they establish a square-root law relationship between the optimal scaling factor and rank, leading to the introduction of LoRA-$\alpha$, which improves performance and simplifies hyperparameter tuning across various tasks.
The scaling factor $\alpha$ in LoRA is not just an auxiliary parameter; it鈥檚 the key to unlocking superior optimization performance that traditional methods overlook.
In Low-Rank Adaptation (LoRA), the scaling factor $\alpha$ is often treated as a mere complement to the learning rate, yet its role in optimization remains poorly understood. In this paper, we reveal that the scaling factor $\alpha$ and the learning rate function differently, with $\alpha$ emerging as the dominant driver of effective optimization, delivering gains that cannot be replicated by learning rate scaling alone. Through the synergy of extensive empirical analysis and a theoretical Signal-Drift framework, we uncover three findings into LoRA's scaling mechanism: First, LoRA's spectral suppression smooths the optimization landscape, rendering standard hyperparameters overly conservative and creating an optimization gap. Second, when leveraging this smoothness to accelerate convergence, $\alpha$ outperforms the learning rate by amplifying the task signal without increasing the drift ratio. Third, the optimal scaling factor follows a sublinear relationship with the rank, well characterized by a square-root law with an unexpectedly large coefficient, revealing the insufficient scaling of existing rank-tied heuristics. Based on these insights, we propose LoRA-$\alpha$, a minimalist framework that restores $\alpha$ to its principled regime, making LoRA compatible with standard small learning rates. Extensive evaluations across diverse tasks demonstrate that LoRA-$\alpha$ consistently improves performance while streamlining hyperparameter search, unleashing the learning potential of LoRA.
