Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces NatScale, a contrast-calibrated framework for mixed-precision quantization of Mixture-of-Experts (MoE) models, addressing the limitation of expert rankings in determining error weighting. By employing a model-specific power transformation to align gradient-based expert weights with an information-theoretic contrast target, NatScale effectively balances precision allocation across experts. Experimental results across three models demonstrate NatScale's superiority in minimizing KL divergence and maximizing top-token agreement compared to five baseline methods, leading to improved perplexity in most settings.
Quantizing MoE models with mixed precision is significantly improved by calibrating expert error sensitivity using an information-theoretic contrast target, outperforming simple ranking-based methods.
Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.