Search papers, labs, and topics across Lattice.
This paper investigates frozen-core adaptation for low-resource fine-tuning of large language models (LLMs) by optimizing a compact $r \times r$ core while fixing the left and right bases of weight matrices. The proposed method, FCCA, utilizes Fisher-whitened cross-covariance to enhance subspace quality and demonstrates significant performance improvements across multiple tasks and model settings. Notably, FCCA achieves an 83.0 macro-average on Qwen2.5-3B, outperforming other basis constructors and showing competitive results compared to larger parameter-efficient methods like LoRA and DoRA, while drastically reducing the number of trainable parameters.
A fixed span can recover most of the benefits of movable low-rank factors in LLM adaptation, achieving superior performance with drastically fewer trainable parameters.
Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through frozen-core adaptation: a calibration pass fixes left and right bases for each weight matrix, and fine-tuning optimizes only an $r\times r$ core. This removes the ability of trainable factors to repair a poor initial span and makes subspace quality directly observable. We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates. Under a matched $r^2$ budget, we compare eight basis constructors on 11 tasks, four model settings, and three seeds. On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks. It ranks first at all three Qwen scales and finishes within 0.13 points of the best method on Llama-3.2-1B. Controlled ablations show gains of 2.7--17.2 points from whitening and identify QR as necessary for stable core optimization in the tested regime. Finally, FCCA comes within 0.32 and 0.23 average points of LoRA and DoRA while optimizing 36.9K rather than roughly 7.4M parameters. These results show that a carefully selected fixed span can recover most of the benefit of movable low-rank factors at a much smaller trainable and optimizer-state cost.
