Search papers, labs, and topics across Lattice.
This paper introduces StatLoRA, a novel statistical inference-based method for rank allocation in low-rank adaptation (LoRA) of large language models, addressing the unequal contributions of model components to downstream tasks. By framing rank allocation as a statistical hypothesis testing problem, StatLoRA utilizes p-values to determine which LoRA components to retain or prune while adhering to a fixed parameter budget. Experimental results demonstrate that StatLoRA outperforms or matches existing methods like vanilla LoRA and AdaLoRA across various natural language tasks, confirming the effectiveness and stability of the proposed approach.
StatLoRA leverages statistical hypothesis testing to optimize rank allocation in LoRA, achieving superior performance while maintaining efficiency.
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization. Existing adaptive rank methods address this problem mainly through carefully designed importance scores constructed from gradient-derived sensitivity and uncertainty measures, without an explicit statistical interpretation. In this paper, we formulate LoRA rank allocation as a statistical hypothesis testing problem and propose StatLoRA, a statistical inference-based rank allocation method. StatLoRA associates each LoRA component with a test statistic and uses estimated p-values to determine which components should be retained or pruned under a prescribed rank budget. The proposed testing procedure is supported by our central limit theory for stochastic optimizer trajectories. In particular, we establish asymptotic normality for a broad class of commonly used optimizers in deep learning, including AdamW, and derive the corresponding asymptotic distributions for the proposed component scores used in hypothesis testing. We evaluate StatLoRA on LoRA fine-tuning of DeBERTaV3-base, BART-Large, and Qwen2.5-7B across natural language understanding, natural language generation, and question answering tasks. Experiments show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and IGU-LoRA under matched rank budgets. Sensitivity analyses and empirical diagnostics further support the stability of the proposed hypothesis-testing-based allocation rule and provide empirical evidence for the asymptotic theory of component scores.
