Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces HiP-LoRA, a novel framework designed to enhance low-rank adaptation of foundation models while mitigating the spectral interference that often leads to catastrophic forgetting. By leveraging cached singular value decomposition (SVD), HiP-LoRA separates updates into principal and residual channels, allowing for a balance between preserving pretrained capabilities and enabling task-specific adaptability. Experimental results on Llama-3.1-8B show that HiP-LoRA significantly reduces pretraining degradation and improves performance in interference-sensitive tasks compared to existing methods.
HiP-LoRA cuts pretraining degradation in half while enabling robust multi-adapter merging, transforming how we approach low-rank adaptation in foundation models.
Adapting foundation models under resource budgets relies heavily on Parameter-Efficient Fine-Tuning (PEFT), with LoRA being a standard modular solution. However, LoRA suffers from spectral interference. Low-rank updates often concentrate energy on the leading singular directions of pretrained weights, perturbing general capabilities and causing catastrophic forgetting and fragile multi-adapter merging. To resolve this, we propose HiP-LoRA, a spectrum-aware adaptation framework. Utilizing the cached singular value decomposition (SVD) of pretrained layers, HiP-LoRA decomposes updates into two channels: a principal channel within the dominant singular subspace, and a residual low-rank channel in the orthogonal complement. A singular-value-weighted stability budget on the principal channel continuously balances pretrained behavior preservation with task-specific plasticity. Experiments on Llama-3.1-8B demonstrate that under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail, robustly outperforming baselines in interference-sensitive tasks like continual tuning and knowledge editing.