Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critiques traditional post-hoc merging methods for multi-task large language models (LLMs), which often lead to task interference and information loss. It introduces METIS, a novel iterative many-shot merging protocol that employs loss-gap balancing and consensus-based masking to enhance performance across tasks. The results demonstrate that METIS significantly improves the performance of the worst-performing task, effectively mitigating information erasure compared to conventional methods.
Iterative many-shot merging can drastically reduce information erasure in multi-task LLMs, leading to better performance on challenging tasks.
Model merging has become a practical post-training strategy for building a single multi-task large language model (LLM) by combining multiple task-specialized models. However, most existing approaches rely on post-hoc merging, in which task-specific models are merged only once after training. This one-shot aggregation often suffers from task interference, leading to information erasure across individual tasks. In this work, we show that replacing post-hoc merging with an iterative many-shot merging protocol is effective in improving multi-task performance. Building on this insight, we propose METIS, Mitigating Erasure from Task Interference for Stable many-shot merging. METIS is a loss-aware many-shot merging method that addresses information erasure in post-hoc merging through task-wise loss-gap weighting and consensus-based masking. Notably, METIS exhibits significant performance improvement on the worst-performing task, effectively mitigating information erasure. (Project page: https://imkyungjin.github.io/METIS/)