Search papers, labs, and topics across Lattice.
This paper introduces CoMerge, a novel framework for multi-task model merging that addresses parameter interference by reformulating the merging process as a preference optimization problem. By leveraging the weaknesses of naive merging methods as hard negative samples, CoMerge employs a self-supervised strategy to refine merging coefficients, effectively preserving task-specific capabilities. Experimental results demonstrate that CoMerge achieves an impressive average normalized performance of 0.9968 on MergeBench, significantly outperforming existing model-merging approaches, particularly in conflict-sensitive tasks.
CoMerge not only preserves task-specific capabilities but also achieves near-optimal performance with just 1,445 scalar coefficients, challenging the need for full-parameter fine-tuning.
Model merging provides an efficient paradigm for constructing multi-task large language models (LLMs) without full model retraining, yet it remains challenged by parameter interference. While existing methods aim to preserve the capabilities of individual expert models and mitigate interference, they generally do not directly learn from the potentially degraded behaviors exposed by naive merging. In this paper, we propose a conflict-driven preference optimization framework for model merging (CoMerge), which reformulates model merging as a preference optimization problem. The approach utilizes a self-supervised, conflict-driven strategy that leverages the defects of naive merging methods (e.g., task arithmetic) as hard negative samples to construct preference pairs without external annotations. By applying preference optimization to refine lightweight, tensor-wise merging coefficients, CoMerge enables the model to mitigate parameter-space conflicts while preserving task-specific capabilities. Extensive experiments show that CoMerge achieves an average normalized performance of 0.9968 on MergeBench, outperforming all evaluated data-free and data-driven model-merging baselines. Furthermore, on Llama-3.1-8B-Instruct, CoMerge yields marked improvements on conflict-sensitive tasks such as instruction following and safety, while remaining highly competitive with full-parameter fine-tuning despite optimizing only 1,445 scalar coefficients.
