Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper disentangles the choices of subspace and factor freezing in spectral LoRA variants by isolating the impact of freezing input (A) versus output (B) factors across different singular value bands. The key finding is that freezing the input factor A, particularly on top singular directions, yields a significant performance advantage over freezing B at low ranks (e.g., rank 2), even outperforming comparable-budget free LoRA. This work extends understanding of factor asymmetry in LoRA by demonstrating its dependence on spectral placement, rank, and training conditions.
Freezing the input factor A in LoRA, especially on top singular directions, unlocks substantial performance gains over freezing the output factor B, even at rank 2.
Spectral variants of low-rank adaptation (LoRA) choose both a subspace and which factor to freeze. We separate these choices by freezing the input factor A or output factor B on the top or bottom singular directions of pretrained weights, with learning rates selected separately. At rank 2, the same-band advantage of freezing A is larger than either within-factor band difference on all four task-model pairs with complete comparisons. Freezing B also trails comparable-budget free LoRA by 8-18 percentage points on five pairs spanning a formatting task and OpenBookQA. The A-frozen advantage persists in a single-GPU-model replication and within individual MLP module groups, including controls with equal or greater trainable counts for B frozen, and when A is frozen on a random orthonormal basis. The factor contrast weakens with rank. On OpenBookQA / Qwen2.5-1.5B at rank 16, PEFT's MiCA implementation trails comparable-budget LoRA by 3.08 points under a shared training recipe transferred from the MiCA paper. A trained oracle output subspace largely removes the low-rank deficit; partial warm-up gains recur across three direction seeds. The factor-versus-band ordering is descriptive; an approximate multiplicity audit weakens several earlier significance claims. These results extend known factor asymmetry by showing how its magnitude depends on spectral placement, rank and training conditions.