Search papers, labs, and topics across Lattice.
This paper introduces RoleMerge, a novel method for merging experts in mixture-of-experts vision-language models (MoE-VLMs) that emphasizes phase-conditioned expert roles rather than relying on global routing statistics. By constructing each expert's Routing Role Profile (RRP) based on phase-normalized routing data, RoleMerge effectively preserves the distinct functions of experts across different phases of inference, leading to improved model performance. The method outperforms traditional expert-merging techniques, achieving up to 9.6% relative improvement in macro-average performance across various benchmarks while maintaining expert retention ratios.
Merging experts based on their phase roles can enhance MoE-VLM performance by up to 9.6%, challenging the effectiveness of traditional global aggregation methods.
Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool. Training-free expert merging reduces this burden, and many routing-based methods aggregate routing statistics across all tokens to determine merge compatibility. However, MoE-VLM inference is phase-structured: image-context tokens carry visual content, question tokens specify the query, and answer tokens produce the output, with different counts and routing distributions. Because image-context tokens are far more numerous, global aggregation can overemphasize image-context processing and obscure phase-conditioned expert roles, making experts serving different phases appear interchangeable and degrading model performance. We therefore argue that MoE-VLM expert merging should preserve phase-conditioned expert roles, judging compatibility by how experts serve different phases rather than globally aggregated routing statistics. Based on this view, we propose RoleMerge, a training-free method that constructs each expert's Routing Role Profile (RRP) from phase-normalized routing statistics, capturing its relative phase preference. Guided by expert-phase information loss, RoleMerge merges experts with compatible profiles and their corresponding router entries while preserving answer-decoding expert distinctions. Experiments on three models and multiple benchmarks show that RoleMerge preserves more of the full model's performance than alternative expert-merging methods at matched expert-retention ratios, with relative improvements of up to 9.6 percent in six-task macro-average performance. These results validate phase-conditioned expert roles as a more effective basis than global routing aggregation for MoE-VLM expert merging.
