Search papers, labs, and topics across Lattice.
This paper investigates the performance of low-rank adaptation (LoRA) methods in the context of Reinforcement Learning with Verifiable Rewards (RLVR), revealing that certain initialized variants, PiSSA and MiLoRA, can underperform compared to standard LoRA. The authors introduce geometry-preserving orthonormal initialization, which theoretically minimizes the gap between LoRA outcomes and full fine-tuning, leading to the development of two new variants, RLPO and RLMO. Experimental results demonstrate that this new initialization approach stabilizes training and significantly enhances performance on mathematical reasoning benchmarks compared to standard LoRA and its variants.
Orthonormal initialization can transform low-rank adaptation in RLVR, outperforming established methods and stabilizing training dynamics.
Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm. However, their efficacy and behavior under Reinforcement learning with verifiable rewards (RLVR) are less well understood. In particular, two structurally initialized LoRA variants, PiSSA and MiLoRA, which outperform standard LoRA under SFT, can underperform standard LoRA under RLVR and may even exhibit training instability. These observations suggest that how to initialize the low-rank matrices in RLVR remains unclear. In this work, we develop a theoretical analysis of LoRA in RLVR, showing that orthonormal initialization achieves the minimal gap between LoRA outcome and that of full fine-tuning. Guided by this insight, we propose geometry-preserving orthonormal initialization for low-rank adaptation in RLVR, leading to two new variants, RLPO and RLMO. Experiments on mathematical reasoning benchmarks show that the proposed orthonormal initialization stabilizes RLVR training and outperforms standard LoRA, contrasting with PiSSA and MiLoRA. Finally, our unified analysis for LoRA initialization also explains why PiSSA and MiLoRA can underperform in RLVR, which may be of independent interest. Code and checkpoints are publicly available at https://github.com/Richard-ZZZ/geometry-preserving-orthonormal-init-rlvr.
