Search papers, labs, and topics across Lattice.
This paper introduces Recover Task eXpert (ReTeX), a framework designed to mitigate parameter interference in multi-task model merging by predicting additive offsets that recover individual task-expert performance from a single merged model. By modeling parameter perturbations as affine transformations and employing a router-free task identifier based on SVD subspace signatures, ReTeX achieves over 95% recovery of individual-expert performance across vision and NLP tasks. Additionally, the framework demonstrates the ability to adaptively interpolate expert knowledge for out-of-distribution tasks, enhancing generalization capabilities significantly.
Recovering over 95% of individual-expert performance from a single merged model could revolutionize multi-task learning efficiency.
Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suffers from parameter interference. While dynamic merging models aim to bridge this gap, many works rely on the costly storage and loading of redundant expert components at inference. In this work, from the perspective of task expert, we view parameter interference as parameter perturbation introduced to each expert during merging process. We show that such parameter perturbations can be modeled as affine transformation, which can be approximated as additive offsets. Motivated by these, we propose Recover Task eXpert (ReTeX), a framework that predicts those offsets, in order to undo parameter interference and recover task-expert performance from a single merged checkpoint. To recover the appropriate expert when task identity is unknown, we introduce a router-free task identifier based on SVD subspace signatures computed offline before inference. At inference, the identifier selects the task whose subspace yields the smallest projection residual for a given input. As a result, ReTeX recovers over 95% of individual-expert performance in both vision and NLP domains, while significantly improving generalization to unseen tasks. Crucially, we also show that the parameter offset prediction leads to emergent adaptive interpolation of expert knowledge for out-of-distribution (OOD) tasks. ReTeX adaptively interpolates seen expert knowledge to handle unseen tasks. Our code is available at https://github.com/BAIKLAB/ReTeX
