Search papers, labs, and topics across Lattice.
This study critically evaluates the validity of using population-level statistics to infer causal importance of experts in Mixture-of-Experts (MoE) models, specifically in the context of pruning. By conducting a token-level interventional audit across three MoE architectures, the authors find that no observational metrics reliably predict causal expert importance, with effect sizes consistently low. The findings challenge the conventional reliance on summary statistics for making interventional claims, highlighting that existing pruning methods operate effectively due to inherent redundancy rather than accurate expert identification.
Observational metrics fail to predict causal expert importance in Mixture-of-Experts models, undermining common pruning practices.
Interpretability methods routinely use population-level summary statistics over observed model behaviour to license claims about the effects of targeted interventions on specific computations; in Pearl's terms, they treat rung-1 associational evidence as if it supported rung-2 interventional conclusions, a move whose validity is rarely tested. We examine one concrete instance: the use of routing statistics in Mixture-of-Experts (MoE) pruning, where utilization rates, activation norms, and routing weight distributions are treated as predictors of which experts can be removed without functional cost. A token-level interventional audit across three high-redundancy MoE architectures (OLMoE-1B-7B-0924, Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite) finds no observational metric predicts causal expert importance after multiple-comparison correction in any model, with effect sizes below Cohen's $d = 0.17$ across all 60 metric-layer combinations. A per-token routing weight control rules out insufficient power, recovering a single Bonferroni-significant signal at OLMoE's final MoE layer ($d = +0.231$, $p = 0.0013$). Existing pruning methods succeed in this regime not by identifying dispensable experts but because early-layer redundancy renders most selection criteria interchangeable. Our results provide an explicit counterexample to the common inferential step from population-level observational summaries to token-level interventional claims about expert importance, and illustrate how interventional audits can calibrate the evidential standards for interpretability claims.
