Search papers, labs, and topics across Lattice.
This paper formalizes the logit shift induced by LoRA using a first-order Fr茅chet approximation around the base model's trajectory. It demonstrates that the effect of multi-layer LoRA can be decomposed into a linear summation of layer-wise contributions, plus a higher-order remainder term representing inter-layer coupling. This decomposition provides a more granular understanding of how LoRA modifies model behavior.
LoRA's seemingly simple parameter updates can be precisely dissected into layer-specific effects, revealing the underlying mechanisms of its influence on model outputs.
This technical note provides a first-order formalisation of the logit shift and fact-margin change induced by Low-Rank Adaptation (LoRA). Using a first-order Fr\'echet approximation around the base model trajectory, we show that the multi-layer LoRA effect can be decomposed into a linear summation of layerwise contributions and a higher-order remainder term representing inter-layer coupling.
