Search papers, labs, and topics across Lattice.
This paper introduces a retraction-free optimization algorithm for the Stiefel manifold, addressing the inefficiencies of existing methods that require orthonormalization or careful tuning of step sizes. By leveraging the properties of the quadratic penalty function and the proximal smoothness of the manifold, the authors establish global convergence guarantees with optimal iteration complexities. The proposed Manifold-LoRA framework reformulates low-rank adaptation for large language models, showing significant improvements in training efficiency and downstream performance through numerical experiments.
A novel retraction-free optimization method achieves global convergence on the Stiefel manifold, drastically improving LoRA fine-tuning efficiency for large language models.
Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method.
BAIR
Tsinghua AI