Search papers, labs, and topics across Lattice.
This paper introduces NA-LoRA, a novel approach to low-rank adaptation (LoRA) that addresses selection misalignment in self-gated Transformer feed-forward networks. By incorporating a nonlinearity-aware principle, NA-LoRA optimally allocates capacity to gate channels based on their responsiveness and shapes the temporal evolution of selection, leading to enhanced performance in fine-tuning tasks. Experimental results demonstrate that NA-LoRA outperforms standard LoRA and competes favorably with other parameter-efficient fine-tuning methods across various benchmarks.
NA-LoRA reveals that adapting low-rank updates with a focus on gate channel responsiveness can significantly enhance model fine-tuning performance.
Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks. In gated FFNs, a low-rank residual can change not only projected features but also the nonlinear selection weights that determine which channels contribute to the output. We formalize this effect as selection misalignment and connect it to the local effective homogeneity of self-gated activations. This motivates a nonlinearity-aware principle for parameter-efficient fine-tuning: low-rank updates should allocate capacity to gate channels whose nonlinear states remain responsive and should shape the temporal evolution of selection. We propose NA-LoRA, a training-only method with two lightweight mechanisms: a derivative-based temporal-importance mask for gate-related LoRA updates and an activation-specific step-scaling rule when a meaningful coarse effective-homogeneity partition is available. NA-LoRA adds no auxiliary loss and incurs no inference-time overhead. Experiments on language-model fine-tuning and vision-language transfer benchmarks show that NA-LoRA consistently improves over vanilla LoRA and is competitive with or better than strong PEFT variants.
