Search papers, labs, and topics across Lattice.
This paper identifies that the performance degradation in merging separately trained LoRA adapters is primarily due to interference from the output-side matrix B, which relies on a limited set of shared directions. To address this, the authors introduce Pico, a data-free calibration method that downscales these shared directions in B before merging, thereby preserving task-specific information from matrix A. The implementation of Pico across various benchmarks results in a significant accuracy improvement of 3.4-8.3 points, outperforming traditional methods and even fully trained LoRA adapters on all task data.
Merging LoRA adapters can actually enhance performance when treating their matrices separately, with Pico achieving up to 8.3 points higher accuracy than existing methods.
Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update 螖W = BA as a single object and do not distinguish the two LoRA matrices. We show that the main source of LoRA merge interference comes from the output-side matrix B. Across tasks, B repeatedly uses a small set of shared directions, while A remains much more task-specific. As a result, the merged adapter overemphasizes these shared directions, and task-specific information is lost. We propose Pico (Pre-merge interference calibration in output-space), a data-free method that calibrates B before merge by downscaling over-shared directions and then rescaling the merged update. Pico plugs directly into existing merging methods such as Task Arithmetic, TIES, and TSV-M. Across eight different benchmarks from math, coding, finance, and medical domains, Pico improves average accuracy by 3.4-8.3 points over the corresponding base method and achieves the best overall average performance. Pico also enables merged adapters to outperform the LoRA trained with all task data. These results show that LoRA merging works better when the two LoRA matrices are treated separately.
