Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MoBiE, a novel post-training quantization framework specifically designed for Mixture-of-Experts (MoE) LLMs, addressing challenges like cross-expert redundancy and quantization-induced routing shifts. MoBiE employs joint SVD decomposition, integrates global loss gradients into Hessian metrics for importance estimation, and uses an input null space guided error constraint to minimize routing distortion. Experiments on models like Qwen3-30B-A3B show MoBiE significantly improves perplexity (52.2%), zero-shot performance (43.4%), and inference speed (2x) compared to existing binarization methods, without increasing storage.
MoBiE slashes the cost of running Mixture-of-Experts models, achieving 2x speedups and massive accuracy gains by intelligently binarizing weights *without* adding storage overhead.
Mixture-of-Experts (MoE) based large language models (LLMs) offer strong performance but suffer from high memory and computation costs. Weight binarization provides extreme efficiency, yet existing binary methods designed for dense LLMs struggle with MoE-specific issues, including cross-expert redundancy, task-agnostic importance estimation, and quantization-induced routing shifts. To this end, we propose MoBiE, the first binarization framework tailored for MoE-based LLMs. MoBiE is built on three core innovations: 1. using joint SVD decomposition to reduce cross-expert redundancy; 2. integrating global loss gradients into local Hessian metrics to enhance weight importance estimation; 3. introducing an error constraint guided by the input null space to mitigate routing distortion. Notably, MoBiE achieves these optimizations while incurring no additional storage overhead, striking a balance between efficiency and model performance. Extensive experiments demonstrate that MoBiE consistently outperforms state-of-the-art binary methods across multiple MoE-based LLMs and benchmarks. For example, on Qwen3-30B-A3B, MoBiE reduces perplexity by 52.2$\%$, improves average zero-shot performance by 43.4$\%$, achieves over 2 $\times$ inference speedup, and further shortens quantization time. The code is available at https://github.com/Kishon-zzx/MoBiE.