Search papers, labs, and topics across Lattice.
NeuronMoE is introduced to improve multilingual LLM extension by allocating MoE experts based on cross-lingual neuron diversity, moving beyond layer-level similarity. By analyzing language-specific neurons across transformer components, NeuronMoE optimizes expert allocation per layer. Experiments on Llama-3.2-3B for Greek, Turkish, and Hungarian show that NeuronMoE achieves a 40% parameter reduction while maintaining performance compared to LayerMoE.
NeuronMoE slashes multilingual LLM parameter counts by 40% without sacrificing performance, by cleverly allocating experts based on neuron-level language specialization rather than blunt layer-level assignments.
Extending large language models to low-resource languages is essential for global accessibility, but training separate models per language is prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by adding sparse language-specific parameters, but determining how many experts each layer needs remains an open question. Current approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons. We propose $\textbf{NeuronMoE}$, a method that analyzes language-specific neurons across all transformer components to guide expert allocation per layer based on empirically measured cross-lingual neuron diversity. Applied to Llama-3.2-3B for low-resource languages (Greek, Turkish, and Hungarian), this approach achieves approximately 40% average parameter reduction while matching the performance of the LayerMoE baseline. We find that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers. This reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.
