Search papers, labs, and topics across Lattice.
This study investigates the impact of adapter capacity on factual acquisition and retention in parameter-efficient fine-tuning, specifically comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) across various ranks. The findings reveal a clear acquisition-retention frontier, where lower ranks of QLoRA maintain out-of-domain performance but acquire fewer facts, while higher ranks enhance same-fact paraphrase generalization at the cost of unrelated benchmark performance. Notably, FFT serves as a conservative baseline, effectively retaining general capabilities but failing to achieve optimal factual acquisition levels, highlighting the nuanced trade-offs in fine-tuning strategies.
Adjusting adapter rank in QLoRA reveals a critical trade-off between factual acquisition and retention of unrelated capabilities, challenging assumptions about parameter-efficient fine-tuning.
Parameter-efficient fine-tuning is often assumed to preserve pretrained capabilities because it updates only a small number of parameters. We show that this assumption depends strongly on adapter capacity. We study factual acquisition in a controlled OpenStreetMap-derived benchmark where Qwen3-4B must acquire anonymized geographic associations while retaining unrelated capabilities. Comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) at ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition--retention frontier. Low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at an increasing cost in performance on unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime. Distributional, weight-space, and spectral diagnostics mirror this behavioral trade-off, with higher-rank QLoRA moving farther from the pretrained model. A separate math adaptation experiment shows a weaker frontier, suggesting that the effect is most pronounced when adaptation must install new factual associations rather than reinforce skills already supported by pretraining. Code and data are available at https://github.com/zhngstl/new_facts_forgetting.
