Search papers, labs, and topics across Lattice.
This paper addresses the challenge of efficiently merging computer vision models fine-tuned for different tasks with heterogeneous, trainable decoders. They introduce a "task alignment proxy" to estimate the quality of merged models without expensive downstream training, enabling rapid hyperparameter selection. Experiments demonstrate the proxy's effectiveness in multi-task vision scenarios, significantly speeding up the merging process while maintaining performance.
Merging vision models for diverse tasks just got way faster: a simple "task alignment" score lets you pick the best merge without expensive retraining.
Efficiently merging several models fine-tuned for different tasks, but stemming from the same pretrained base model, is of great practical interest. Despite extensive prior work, most evaluations of model merging in computer vision are restricted to image classification using CLIP, where different classification datasets define different tasks. In this work, our goal is to make model merging more practical and show its relevance on challenging scenarios beyond this specific setting. In most vision scenarios, different tasks rely on trainable and usually heterogeneous decoders. Differently from previous studies with frozen decoders, where merged models can be evaluated right away, the non-trivial cost of decoder training renders hyperparameter selection based on downstream performance impractical. To address this, we introduce the task alignment proxy, and show how it can be used to speed up hyperparameter selection by orders of magnitude while retaining performance. Equipped with the task alignment proxy, we extend the applicability of model merging to multi-task vision models beyond CLIP-based classification.
