Search papers, labs, and topics across Lattice.
The paper investigates the spectral properties of LoRA updates, finding that task-relevant information is often concentrated in a small subset of singular directions, leading to inefficient parameter allocation. To address this, they propose Spectral Surgery, a training-free refinement method that uses SVD to decompose the LoRA update and reweights singular values based on gradient-estimated sensitivity. Experiments on Llama-3.1-8B and Qwen3-8B across four benchmarks demonstrate consistent performance gains by adjusting only a small number of scalar coefficients, showcasing the effectiveness of post-hoc spectral refinement.
LoRA adapters are spectrally inefficient, but you can boost their performance by up to 4.4 points with a training-free SVD reweighting trick that adjusts just 1,000 scalar values.
Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace, yet how this limited capacity is allocated within a trained adapter remains unclear. Through a geometric and empirical study across multiple tasks and backbones, we find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions, while many remaining components are neutral or detrimental, motivating post-hoc refinement within the learned subspace. We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed. Across Llama-3.1-8B and Qwen3-8B on four benchmarks, Spectral Surgery yields consistent gains (up to +4.4 points on CommonsenseQA and +2.4 pass@1 on HumanEval) by adjusting only $\approx 1{,}000$ scalar coefficients. These results demonstrate that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.
