Search papers, labs, and topics across Lattice.
This paper addresses the challenge of task interference in model merging by introducing a sparse-representation-based framework that utilizes Sparse Autoencoders (SAEs) for high-dimensional feature disentanglement. By projecting task vectors into a sparse feature space, the method effectively isolates useful task directions from interfering components, leading to improved performance in multi-task generalist models. Experimental results on Qwen2.5 models show that this approach consistently outperforms existing baselines, achieving a notable 2.78% improvement in a highly conflicting task setting.
Merging models without losing performance is possible by disentangling task-specific features in high-dimensional space, yielding significant improvements even in conflicting scenarios.
Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from \textit{superposition}, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78\% improvement over the strongest baseline.
