Search papers, labs, and topics across Lattice.
This paper introduces DistMoE, a mixture-of-experts (MoE) framework designed for distributed visual instruction tuning of multimodal large language models (MLLMs) without requiring centralized data access. By incorporating client-specific private feedforward network (FFN) experts into the language decoder and employing a public-anchored expert composition stage, DistMoE effectively mitigates client-specific drift and enables seamless integration of domain-specific knowledge. Experimental results demonstrate that DistMoE achieves competitive performance across various visual-language benchmarks while allowing for flexible expert reuse and modular control over client-specific information.
DistMoE enables MLLMs to adapt to diverse visual-language domains without centralized data, achieving competitive performance while preserving client-specific knowledge.
Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, we propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal to acquire domain-specific knowledge. However, independent expert training causes the private FFNs to learn representation of different scale and magnitudes, making merging the experts difficult. To reduce client-specific drift, we introduce a public-anchored expert composition stage that updates only routers and lightweight private projection adapters on a mix of local client data and public data, via an isotropic regularization loss, therefore making it cross-client rehearsal-free composition. During inference, DistMoE performs modular routing over public and private experts, enabling token-wise domain composition without explicit domain labels. Experiments across diverse visual-language benchmarks show that DistMoE enables flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Codes are available at https://github.com/mainaksingha01/DistMoE.
