Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Value-and-Structure Routing Alignment for Quantization (VSRAQ), a novel post-training quantization method tailored for Mixture-of-Experts (MoE) models that addresses the challenges of routing instability during quantization. By combining value alignment and structure alignment, VSRAQ effectively preserves expert-selection behavior, ensuring that quantization does not disrupt the model's computation paths. Experimental results demonstrate that VSRAQ significantly enhances expert-selection consistency and outperforms both reconstruction-only and router-aware baselines without adding inference-time overhead.
VSRAQ ensures that quantization doesn't compromise expert selection in MoE models, preserving performance while streamlining deployment.
Mixture-of-Experts (MoE) models scale foundation models efficiently by activating only a subset of experts for each token, but their large number of expert parameters still makes quantization essential for practical deployment. Unlike dense models, however, MoE models are sensitive to routing instability: small quantization-induced perturbations can change the top-$k$ expert selection, altering the computation path and degrading model quality. We propose Value-and-Structure Routing Alignment for Quantization (VSRAQ), a MoE-specific post-training quantization objective that preserves pre-quantization expert-selection behavior under quantization. VSRAQ combines two complementary objectives that jointly preserve expert-selection behavior: value alignment, which matches routing-relevant logits or scores, and structure alignment, which preserves expert ordering and top-$k$ decision boundaries. By maintaining routing consistency, VSRAQ reduces quantization-induced degradation without introducing any inference-time overhead and can be integrated into existing quantization frameworks. Experiments on recent MoE foundation models show that VSRAQ improves expert-selection consistency and consistently outperforms reconstruction-only and router-aware baselines.