Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper identifies an exploration dilemma in modern Mixture-of-Experts (MoE) architectures, where fine-grained expert segmentation and shared experts create conflicting routing needs. Specifically, shared experts reduce token representation variability, hindering the exploration required by the combinatorially enlarged routing space of fine-grained segmentation. MirrorMoE resolves this by adaptively suppressing unnecessary experts, thereby shrinking the exploration space and unlocking the full potential of fine-grained MoE designs, leading to consistent performance gains across scales.
Shared experts in MoEs paradoxically limit the very exploration fine-grained segmentation needs, but MirrorMoE unlocks this potential by adaptively pruning the routing space.
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.