Search papers, labs, and topics across Lattice.
This paper introduces UniMoMo, a post-training compression framework designed to convert large recommendation models with sparse mixture-of-experts (MoE) layers into smaller, more efficient versions without the need for additional online modules. By formulating the problem as constrained graph coarsening, UniMoMo groups experts based on their functional similarity, leveraging an unlabeled calibration set to assess expert responses to shared states. The approach achieves impressive performance retention, with NDCG@10 ratios between 98.36% and 104.24% and speedups of 1.47x to 2.21x across various configurations, demonstrating its effectiveness in deploying MoE models under strict expert budgets.
Achieving over 99% performance retention while significantly accelerating large recommendation models by intelligently merging experts based on functional similarity.
Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a smaller standard MoE under an explicit expert budget, without adding a compression-specific online module. To address this, we introduce UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem. Rather than relying on parameter distance, UniMoMo groups experts based on their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. To prevent performance degradation, we introduce a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure. Across Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the final four-expert checkpoints obtain source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% and measured A100 speedups of 1.28$\times$--1.63$\times$. An aggressive two-expert, top-1 operating point obtains ratios of 98.36%--104.24% and speedups of 1.47$\times$--2.21$\times$. These endpoint results evaluate the complete conversion-and-adaptation workflow and show that a trained recommendation MoE can be exported at multiple serving budgets.
