Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
We investigate personalizing Mixture-of-Experts (MoE) LLMs by leveraging their sparse routing to reduce polysemanticity and enable localization of personalization components. By training crosscoders on paired activations from personalized and non-personalized MoE models, we identify interpretable latents associated with specific experts encoding distinct personalization features. Causal interventions confirm these latents are sufficient to recover personalized behavior and that boosting relevant experts shifts non-personalized models towards personalization.
Specialized experts in MoE LLMs, not dense model neurons, are the key to interpretable and controllable personalization.
Personalization of Large Language Models (LLMs) is increasingly important for user-centric applications, yet prior methods mainly adapt dense LLMs, whose polysemantic neurons, each encoding multiple unrelated concepts, hinder localizing the components behind personalization, and thus understanding and controlling it. We propose to personalize Mixture-of-Experts (MoE) LLMs, which sparsely route each token to a few specialized experts, reducing polysemanticity and enabling such localization. We adapt three open-weight MoE LLMs on a large-scale personalization dataset and train crosscoders on paired activations of the personalized and non-personalized models to learn interpretable personalized latents, from which we identify the experts encoding distinct personalization features. The personalized MoE LLMs perform competitively with dense LLMs while activating only a fraction of their parameters, exhibit lower polysemanticity than dense LLMs with similar active parameters, and, within the same model family, yield personalized latents that are less shared with the non-personalized model. Causal interventions show that these latents suffice to recover the personalized model's behavior and that boosting their associated experts moves the non-personalized model toward it, revealing distinct routing paths for different personalization features.