Search papers, labs, and topics across Lattice.
This paper introduces LoRA-TSD, an innovative optimizer for low-rank adaptation (LoRA) that optimally navigates the geometry of low-rank weight changes by treating each update as a tangent vector on the fixed-rank matrix manifold. By employing spectral-norm steepest-descent steps within this tangent space and utilizing a cost-effective retraction method, LoRA-TSD significantly reduces computational overhead compared to traditional methods. The results demonstrate that LoRA-TSD not only surpasses all existing LoRA optimizers across multiple benchmarks but also provides the first global convergence guarantees for both LoRA-Pro and itself, ensuring effective training dynamics.
LoRA-TSD achieves superior performance over all existing LoRA optimizers while offering the first global convergence guarantees for low-rank adaptations.
Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization. The step avoids expensive operations on full weight matrices, and its retraction is up to $2.8\times$ cheaper than the truncated-SVD retraction used by prior manifold methods. We prove that the Frobenius-norm version of our surrogate recovers LoRA-Pro, and we identify the tangent-projected gradient, the Riemannian gradient of the manifold, as the stationarity measure natural to LoRA training and computable from the factor gradients alone. Under this measure we give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero. Across six commonsense and natural-language-inference benchmarks with Llama-3.2-1B, Llama-3.1-8B and Qwen3-32B, LoRA-TSD outperforms every competing LoRA optimizer and stays robust to the adapter rank. Code is available at https://github.com/brain-lab-research/LoRA-TSD.
