Search papers, labs, and topics across Lattice.
The paper introduces MixtureKit, an open-source framework designed to facilitate the construction, training, and analysis of Mixture-of-Experts (MoE) models using pre-trained or fine-tuned models. MixtureKit implements three MoE methods: Traditional MoE, BTX (fine-grained token routing), and BTS (trainable stitch layers for information exchange). Experiments on multilingual code-switched data demonstrate that BTX-based models built with MixtureKit outperform dense baselines, showcasing the framework's utility.
MixtureKit offers a unified, open-source toolkit that simplifies the creation and analysis of Mixture-of-Experts models, enabling researchers to easily experiment with different routing strategies and visualize expert behavior.
We introduce MixtureKit, a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned models. MixtureKit currently supports three complementary methods: (i) \emph{Traditional MoE}, which uses a single router per transformer block to select experts, (ii) \emph{BTX} (Branch-Train-Mix), which introduces separate routers for each specified sub-layer enabling fine-grained token routing, and (iii) \emph{BTS} (Branch-Train-Stitch), which keeps experts fully intact and introduces trainable stitch layers for controlled information exchange between hub and experts. MixtureKit automatically modifies the model configuration, patches decoder and causal LM classes, and saves a unified checkpoint ready for inference or fine-tuning. We further provide a visualization interface to inspect per-token routing decisions, expert weight distributions, and layer-wise contributions. Experiments with multilingual code-switched data (e.g. Arabic-Latin) show that a BTX-based model trained using MixtureKit can outperform baseline dense models on multiple benchmarks. We release MixtureKit as a practical foundation for research and development of MoE-based systems across diverse domains.
