Search papers, labs, and topics across Lattice.
The paper introduces ACE-Merging, a data-free model merging technique that addresses the challenge of interference between expert models trained on different objectives. It leverages a theoretical analysis showing that the input covariance of each task can be implicitly estimated from parameter differences, enabling effective mitigation of inter-task interference without data access or retraining. ACE-Merging uses this insight to derive a closed-form solution for merging, achieving state-of-the-art performance among data-free methods, with a 4% average improvement over previous methods on GPT-2 across seven tasks.
Forget retraining: ACE-Merging unlocks state-of-the-art model merging by cleverly estimating task covariance from parameter differences alone.
Model merging aims to combine multiple task-specific expert models into a single model while preserving generalization across diverse tasks. However, interference among experts, especially when they are trained on different objectives, often leads to significant performance degradation. Despite recent progress, resolving this interference without data access, retraining, or architectural modification remains a fundamental challenge. This paper provides a theoretical analysis demonstrating that the input covariance of each task, which is a key factor for optimal merging, can be implicitly estimated from the parameter differences of its fine-tuned model, even in a fully data-free setting. Building on this insight, we introduce \acem, an Adaptive Covariance Estimation framework that effectively mitigates inter-task interference. Our approach features a principled, closed-form solution that contrasts with prior iterative or heuristic methods. Extensive experiments on both vision and language benchmarks demonstrate that \acem sets a new state-of-the-art among data-free methods. It consistently outperforms existing baselines; for example, \acem achieves an average absolute improvement of 4\% over the previous methods across seven tasks on GPT-2. Owing to its efficient closed-form formulation, \acem delivers superior performance with a modest computational cost, providing a practical and theoretically grounded solution for model merging.
