Search papers, labs, and topics across Lattice.
The paper introduces D2-LoRA, a parameter-efficient fine-tuning method that combines signed low-rank residual updates with additive and subtractive components and a column-wise projection to maintain norm proximity. This approach achieves state-of-the-art performance on question answering, reading comprehension, and generative tasks with limited data and compute, while preserving algebraic mergeability for zero-latency inference. Experiments demonstrate that D2-LoRA outperforms LoRA and matches or exceeds DoRA, exhibiting improved accuracy, stability, and throughput.
D2-LoRA squeezes out a 2.2% accuracy boost over LoRA using the same number of parameters, proving that smarter adapter architectures can beat brute-force scaling in parameter-efficient fine-tuning.
We systematically investigate the parameter-efficient fine-tuning design space under practical data and compute constraints, and propose D2-LoRA. D2-LoRA achieves 76.4 percent average accuracy across eight question answering and reading comprehension benchmarks using only 5k training samples per task and two epochs, while preserving algebraic mergeability at inference with near-exact numerical equivalence. The method combines signed low-rank residual updates with additive and subtractive components, together with a train-time column-wise projection that keeps each column close to its original norm. After training, the adapter is merged into a single weight matrix, adding zero inference latency. Compared with LoRA, D2-LoRA improves average accuracy by 2.2 percentage points; at matched parameter counts (LoRA rank 2r versus D2-LoRA rank r), the improvement is 1.6 points, indicating gains from architectural design rather than increased parameterization. Compared with DoRA, it matches or exceeds performance on most tasks. Beyond QA and reading comprehension, D2-LoRA improves generative tasks (plus 1.2 ROUGE-L and plus 1.1 percent win rate) and shows 36 percent lower training volatility. The merge preserves numerical fidelity (mean gap about 0.03 percentage points) and recovers about 1.91x evaluation throughput. Training overhead is 19 percent, comparable to DoRA, and decreases with longer input sequences. We provide a geometric analysis explaining how the projection stabilizes training, together with ablation studies isolating the contribution of each design component.
