Search papers, labs, and topics across Lattice.
This paper establishes theoretical bounds on the approximation error for low-rank adaptation (LoRA) updates in Transformer attention, focusing on the rank selection's impact on performance. By analyzing the relationship between the rank of LoRA updates and the Kullback-Leibler error in relation to a fixed pretrained attention head and target attention function, the authors derive both lower and upper error bounds dependent on task-specific conditions. The findings reveal that the required rank for effective adaptation can be significantly lower than previously assumed, particularly in cases of softmax saturation, thereby optimizing the efficiency of LoRA in practical applications.
The required rank for effective low-rank adaptation in Transformer attention can be much smaller than expected, especially under softmax saturation conditions.
Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-$r$ query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to $\psi(\|d\|_2)$, where $d$ is the difference between candidate and target attention scores and $\psi(t)=\min\{t^2,t\}$. We also prove an unconditional upper bound $\min\{\|d\|_2^2/4,\sqrt2\|d\|_2\}$. Under explicit realizability, geometry, and moment conditions, we then bound the best rank-$r$ error between an explicit multiple of $\psi(\sqrt{T_r})$ and $\min\{T_r/4,\sqrt{2T_r}\}$, where $T_r$ is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.
