Search papers, labs, and topics across Lattice.
This paper re-evaluates the training of domain-specialized experts for model merging by introducing a prompt-based adaptation strategy that maintains a fixed backbone to reduce interference during merging. The authors present Dual-Tuned Experts (DTEs), a two-stage training approach that first learns prompts before fine-tuning the vision encoder, leading to experts that are more compatible for merging. Experimental results demonstrate that DTEs enhance the performance of merged models across various CLIP architectures and expert types, outperforming traditional full fine-tuning methods.
Prompt-based adaptation can significantly enhance the merging of specialized models, yielding better performance without the pitfalls of traditional weight merging.
Model merging aims to combine multiple domain-specialized experts trained from a shared foundation model into a single multi-task model. Existing approaches largely focus on improving the merging procedure itself and typically assume experts obtained through full-parameter fine-tuning. In this work, we revisit expert training for model merging. We first show that prompt-based adaptation provides a strong baseline: independently learned prompts can be exploited across tasks while keeping the backbone fixed, avoiding the interference introduced by weight merging. Building on this observation, we introduce Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns prompts and then fine-tunes the vision encoder. This reduces the magnitude of task-specific parameter updates and produces experts with higher merge compatibility. Experiments across multiple CLIP architectures, full fine-tuning, and LoRA experts show that DTEs consistently improve merged performance of standard merging approaches and remain effective even when combining heterogeneous sets of experts.
