Search papers, labs, and topics across Lattice.
This paper introduces Normalized Low-Rank Adaptation (NoRA), a method that enhances the training dynamics of low-rank adaptation (LoRA) by normalizing down-projection matrices during training. The authors demonstrate that this normalization leads to faster convergence, improved performance, and greater stability across various tasks, including pretraining, supervised finetuning, and reinforcement learning. Notably, NoRA achieves these benefits without adding extra trainable parameters or increasing inference-time computation, making it a practical enhancement to existing LoRA techniques.
Normalized Low-Rank Adaptation accelerates training and enhances performance without increasing model complexity or inference costs.
While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.
