Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper investigates the emergent semantic specialization of experts within multimodal Mixture-of-Experts (MoE) architectures, finding that experts develop distinct domain and modality proficiencies without explicit modularity training. Leveraging this intrinsic organization, they introduce ExpertLens, a data-free technique that decodes router weights to identify domain-specific experts. Efficient adaptation is then achieved by selectively fine-tuning these identified experts, matching or exceeding full fine-tuning performance with significantly fewer parameters and faster training times.
Multimodal MoE experts spontaneously develop semantic specializations, enabling data-free identification and highly efficient, targeted adaptation that outperforms LoRA.
Mixture-of-Experts (MoE) architectures scale model capacity through sparse computation, routing each token through only a small subset of experts. In this work, we explore whether this sparsity gives rise to emergent intrinsic organization in multimodal MoEs. We find that experts develop strong semantic specialization across modalities and domains despite not being explicitly trained for modularity. Building on this structure, we introduce ExpertLens, a data-free method that identifies domain-specialized experts directly from pretrained model weights by decoding router weights into semantically meaningful vocabulary tokens. We leverage this specialization for efficient multimodal adaptation by selectively fine-tuning experts relevant to a target domain. Across math, medical, and remote sensing tasks, ExpertLens matches or surpasses full fine-tuning while updating only 21.7 - 47.0% of model parameters and achieving a 4.0x average training speedup, and outperforms LoRA in both adaptation performance and training efficiency. These results show that sparsity introduced for efficiency can give rise to semantic modularity that is directly useful for efficient adaptation.