Search papers, labs, and topics across Lattice.
This study quantitatively analyzes the memorization capacity of low-rank adapters (LoRA) in the context of parameter-efficient fine-tuning, revealing that these adapters write significantly fewer bits into a model than previously assumed. The findings indicate that the capacity of an adapter is influenced more by the location of parameters within the model architecture than by the total number of parameters itself, with a notable increase in capacity when parameters are allocated to the MLP instead of the attention layer. Additionally, the research highlights a concerning trend of privacy leakage correlated with the bits written by adapters, differentiating the memorization behaviors of supervised versus reinforcement learning fine-tuning approaches.
Adapters may store less than half the information expected, with their capacity heavily influenced by parameter placement rather than quantity.
A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it. We put that assumption on a scale. Extending compression-based memorization analysis to the frozen-base setting, we measure directly, in bits, how much a low-rank adapter writes into a model it never changes. The answer is both smaller than full fine-tuning and less lawful than parameter counting would predict. Adapters store a couple of bits per trainable parameter, well short of a full model's budget, but that figure turns less on how many parameters an adapter carries than on where they sit. Move the same parameter budget from attention into the MLP and it holds nearly twice as much; strip the frozen base of its structure and the capacity all but disappears. Applied to realistic fine-tunes of Qwen2.5, the same instrument shows privacy leakage rising with the bits an adapter writes rather than the parameters it nominally has, and it draws a clean line between supervised and reinforcement learning: the secrets that supervised fine-tuning copies down verbatim, an adapter trained on verifiable rewards never records. Measuring what fine-tuning writes, rather than attacking it after the fact, turns a piece of folklore into a quantity one can design against.
CMU ML